[WIP] Juju acceptance testing primer


#1

Background

Juju’s code base includes an acceptance testing system within the juju/acceptancetests directory. It’s called jujupy, which should not be confused with the Twisted implementation of Juju circa 2015.

jujupy is written in Python and is considered to be a dark art somewhat. Of the current team members, @achilleasa, @kelvin.liu and @simonrichardson have the most experience writing tests.

For whatever reason, we decided to build our own tool rather than using something like Cucumber. Although it increases the maintainence burden, that provides a few advantages:

  • no DSL to learn
  • we can use Canonical-centric technologies, such as as LXD

What is an acceptance test?

Acceptance testing—as the term relates to Juju—is isolated black box testing. jujupy runs Juju as if it were a human user on the command line. Every test provisions goes through the bootstrap process, executes whichever steps that the test script defines, then cleans up after itself…

Our tooling does no introspection of the internals of the system. It has no more knowledge of what’s happening than a person sitting at her desk running commands in the shell.

How an acceptance test is run

The standard method for running an acceptance test is via the assess script that lives within juju/acceptancetests. Assuming that you’re there, running a test is straightforward:

./assess upgrade

To run a test within AWS, use the --substrate argument:

./assess --substrate=aws upgrade

./assess carries out multiple

This sets up the environment for running the test, including creating a temporary directory and a LXD container.

During each test, we start from a completely clean environment. The container needs to upgrade itself and juju needs to bootstrap. This makes tests very slow and generates significant logging messages.

Once the test has completed, jujupy will also generate even more logging messages. The intent is for the entire state of the system to be visible, to facilitate debugging.

Writing a new test

There are a few conventions to be followed:

  • name your file assess_{test}.py, where {test} is the name of the test
  • to generate command-line arguments, use jujupy.add_basic_testing_arguments()

The general pattern is enforced by a template script that can

Conventions

Define a parse_args() function that handles command-line argument parsing.

Define a main() method that sets up the bootstrap process and runs test assertions.

Test assertions are defined in functions.

Add an easy to grep logging line when the tests are successful. This makes it much easier to debug later.

log.info("PASS assess_my_test")

TODO: examples and more details

Helper utilities

Several helper functions are provided in acceptancetests/utility.py. To make use of them, import them from your test file. Making use of them increases the consistency of our tests, which provides maintainence benefits as well as other nice side effects such as consistent logging messages.

from utlity import wait_for_port

Unfortunately, few of the functions are well documented. Never fear! Find examples from within the other tests.

Some pointers:

  • run_command()
    Executes a command in the shell, logging what it’s doing along the way.
  • add_model()
    Adds a model to the current controller.

Features of jujupy

TODO: section needs a lot of work

The BootstrapManager.from_args() can take care of bootstrapping.

A ModelClient accepts a path argument to a Juju binary. If this is set to the string FAKE, jujupy will make use of its own “fake” client, rather than using an actual Juju. I still need to investigate what the significance of this is.

Debugging tests

Your test has failed. Now what? Every test generates a lot of output. Before teardown, jujupy inspects the model state and prints everything out very verbosely. Likewise the bootstrap process if recorded in detail. These messages can mask what’s actually happening.

Adding easy-to-grep logging messages that mark the beginning of the test process.

Personal Aside

jujupy and the related files are not written in a style that I’m fond of. It makes heavy use of metaclasses, decorators and other advanced Python features that make using the library difficult for programmers who primarily use Go, and may be unfamiliar with advanced Python.


#2

What are the thoughts around the fact that assess now causes us to edit two places (assess and the actual assess_{test}.py), I’m unsure if that’s going to cause more fragility long term. How about we just pass all the arguments to the assess_{test}.py and let that resolve it.

The greatest strength to assess is setting up the environment, not parsing the arguments…

For example in bash you could do something like the following to pass the arguments, I’m sure you can do the same in python:

#!/bin/sh -e
test_name="./assess_$1.py"
shift 1
$test_name "$@"

#3

assess needs a few of the arguments itself. Additionally, assess does some normalisation of arguments for cases when people have used different arguments to do the same thing.


#4

This impact should be minimal. Most tests don’t require custom arguments and don’t need to touch assess at all.

In my view, the minimal impact that it does have on maintainece is strongly outweighed by the overall productivity gains. The tests are run far more often than they are written.