Background
Juju’s code base includes an acceptance testing system within the juju/acceptancetests
directory. It’s called jujupy
, which should not be confused with the Twisted implementation of Juju circa 2015.
jujupy
is written in Python and is considered to be a dark art somewhat. Of the current team members, @achilleasa, @kelvin.liu and @simonrichardson have the most experience writing tests.
For whatever reason, we decided to build our own tool rather than using something like Cucumber. Although it increases the maintainence burden, that provides a few advantages:
- no DSL to learn
- we can use Canonical-centric technologies, such as as LXD
What is an acceptance test?
Acceptance testing—as the term relates to Juju—is isolated black box testing. jujupy
runs Juju as if it were a human user on the command line. Every test provisions goes through the bootstrap process, executes whichever steps that the test script defines, then cleans up after itself…
Our tooling does no introspection of the internals of the system. It has no more knowledge of what’s happening than a person sitting at her desk running commands in the shell.
How an acceptance test is run
The standard method for running an acceptance test is via the assess
script that lives within juju/acceptancetests
. Assuming that you’re there, running a test is straightforward:
./assess upgrade
To run a test within AWS, use the --substrate
argument:
./assess --substrate=aws upgrade
./assess
carries out multiple
This sets up the environment for running the test, including creating a temporary directory and a LXD container.
During each test, we start from a completely clean environment. The container needs to upgrade itself and juju needs to bootstrap. This makes tests very slow and generates significant logging messages.
Once the test has completed, jujupy
will also generate even more logging messages. The intent is for the entire state of the system to be visible, to facilitate debugging.
Writing a new test
There are a few conventions to be followed:
- name your file
assess_{test}.py
, where{test}
is the name of the test - to generate command-line arguments, use
jujupy.add_basic_testing_arguments()
The general pattern is enforced by a template script that can
Conventions
Define a parse_args()
function that handles command-line argument parsing.
Define a main()
method that sets up the bootstrap process and runs test assertions.
Test assertions are defined in functions.
Add an easy to grep logging line when the tests are successful. This makes it much easier to debug later.
log.info("PASS assess_my_test")
TODO: examples and more details
Helper utilities
Several helper functions are provided in acceptancetests/utility.py
. To make use of them, import
them from your test file. Making use of them increases the consistency of our tests, which provides maintainence benefits as well as other nice side effects such as consistent logging messages.
from utlity import wait_for_port
Unfortunately, few of the functions are well documented. Never fear! Find examples from within the other tests.
Some pointers:
-
run_command()
Executes a command in the shell, logging what it’s doing along the way. -
add_model()
Adds a model to the current controller.
Features of jujupy
TODO: section needs a lot of work
The BootstrapManager.from_args()
can take care of bootstrapping.
A ModelClient
accepts a path argument to a Juju binary. If this is set to the string FAKE
, jujupy
will make use of its own “fake” client, rather than using an actual Juju. I still need to investigate what the significance of this is.
Debugging tests
Your test has failed. Now what? Every test generates a lot of output. Before teardown, jujupy
inspects the model state and prints everything out very verbosely. Likewise the bootstrap process if recorded in detail. These messages can mask what’s actually happening.
Adding easy-to-grep logging messages that mark the beginning of the test process.
Personal Aside
jujupy
and the related files are not written in a style that I’m fond of. It makes heavy use of metaclasses, decorators and other advanced Python features that make using the library difficult for programmers who primarily use Go, and may be unfamiliar with advanced Python.