Users need to be able to roll changes to applications in a safe guided processes that controls the flow such that not all units of an HA application are hit at once. This also allows some manual canary testing and provides control over the flow of changes out to the model.
We propose to allow operators to manage this using model generations.
What is a model generation?
A model generation can refer to the current generation (definition) of the model or a next generation that is being defined and rolled out with control.
Basic use case walk through
A typical use case would be to update a charm revision, update the config for the new charm revision, and to roll that out such that you can validate it behaves correctly before rolling that change to all units. In our example let’s update Keystone.
$ juju add-generation target generation set to next $ juju upgrade-charm keystone Added charm "cs:keystone-285" to the model. $ juju config keystone debug=true
With those changes now awaiting rollout to the model in the next generation the user can push it out to units selectively.
$ juju set-generation keystone/0
Additional changes can be made to the next generation.
$ juju config ceph-mon loglevel=10
Note: this change will be visible to
keystone/0 immediately, but not to the other units.
And these can be pushed out at the application level as well. The user may also specify multiple targets to the
$ juju set-generation ceph-mon keystone/1 keystone/2
Once the changes in the next generation are all pushed out the generation is deemed to be live and it the target generation becomes current.
$ juju set-generation keystone/3 all changes in next generation complete, target generation set to current
Which model operations are tracked via generations?
Not all changes to the model are tracked in this way.
These are the operations that can be pinned to a future generation. This might change over time, but anything outside of those operations are considered out of scope for this specification.
Targeting options for pushing to the next generation
set-generation command takes one or more model targets which may consist of application or unit names. In this way you might update a few related applications, update a select unit of each, and validate your changes are solid.
$ juju add-generation target generation set to next $ juju upgrade-charm keystone Added charm "cs:keystone-285" to the model, queued in next generation. $ juju upgrade-charm percona-cluster Added charm "cs:percona-cluster-270" to the model, queued in next generation. $ juju set-generation keystone/0 percona-cluster/0 *manual validation of the updated charms here*
The user can then push out to the rest of the units by just using the application as the target.
$ juju set-generation keystone percona-cluster all changes in next generation complete, target generation set to current
Making multiple changes to the next generation
set-generation to a unit and later make another change that affects that unit it will be instantly sent out. The idea is that once you
set-generation that unit is tracking the next generation and will get live updates.
$ juju add-generation target generation set to next $ juju config ceph-mon loglevel=1 $ juju set-generation ceph-mon/0 $ juju config ceph-mon loglevel=10
At this point the ceph-mon/0 unit will receive a second config-changed event with the
loglevel value of 10.
Making changes that occur immediately and are not related to future generations
Users may switch between the current and next generations.
$ juju switch-generation current target generation set to current
And now changes go out live as they normally would without any need to push them selectively.
$ juju config ceph-mon loglevel=1
Commands can also be scripted to target generations specifically.
$ juju config --generation=current ceph-mon loglevel=1
Cancelling a next generation
If a next generation is not going to be rolled out you can cancel it using the command
cancel-generation. This will abort anything in the next generation and return the active target to the current generation.
$ juju add-generation target generation set to next $ juju config ceph-mon loglevel=1 $ juju cancel-generation changed dropped and target generation set to current
Note that you cannot cancel a generation with changes that are partially pushed out to only select units. You’ll need to set a value and push to the units involved so that there’s consistency across the units. You can cancel changes that have not been pushed out. Take the following example:
$ juju add-generation target generation set to next $ juju config keystone debug=true $ juju config ceph-mon loglevel=1 $ juju set-generation keystone/0 $ juju cancel-generation ERROR: cannot cancel generation, there are units behind a generation: keystone/1, keystone/2 $ juju set-generation keystone/1 keystone/2
At this point all of the Keystone units have the new config but the ceph-mon change has not been pushed to any units. Since they are consistent we can cancel the changes and return to the current generation.
$ juju cancel-generation changed dropped and target generation set to current
Tracking units not on the up to date generation
When a new generation is added and changes come in status will indicate units that are behind a generation. These units need a
set-generation command to get the latest details and become up to date.
$ juju add-generation target generation set to next $ juju config keystone debug=true $ juju config percona-cluster tuning-level=safest $ juju set-generation keystone/0 percona-cluster/0 $ juju status Model Controller Cloud/Region Version SLA Timestamp default jujudemo google/us-east1 2.4.2 unsupported 13:09:59-04:00 Unit Workload Agent Machine Public address Ports Message keystone/0* blocked executing 0 18.104.22.168 5000/tcp Missing relations: database keystone/1 blocked executing 1 22.214.171.124 5000/tcp [old] Missing relations: database keystone/2 blocked executing 2 126.96.36.199 5000/tcp [old](start) Missing relations: database percona-cluster/0* active idle 3 188.8.131.52 3306/tcp Unit is ready percona-cluster/1 maintenance executing 4 184.108.40.206 [old] Waiting 30 seconds for operation ... percona-cluster/2 maintenance executing 5 220.127.116.11 [old] Waiting 60 seconds for operation ... ...
Note that the units that are on the previous generation are noted as being
[old]. These are targets that need to be updated before the transition to the next generation is complete. Note that the indication is on the unit
Message field regardless of what the changes made are. A config change, charm upgrade, or resources change will all indicate in the same way. The details about what is different can be investigated using the
diff-generation command noted below.
Showing the difference between the current and next generations
Users can see what changes are made to the next generation using the command
diff-generation. It will display only those changes that were made since the generation was added. It will not indicate which have been pushed or not. Only the changes made themselves.
$ juju add-generation target generation set to next $ juju config keystone debug=true $ juju config ceph-mon loglevel=1 $ juju diff-generation applications: keystone: config: debug: true ceph-mon: config: loglevel: 1
<note: this output format should look like an overlay or something that might be in common with diff-bundle work and such>