The goal of this tutorial is to provide some practical information for managing credentials. It tries to not reiterate what is already provided on the Credentials page but approaches the subject from a more practical level where often misunderstood concepts are reinforced and field-reported scenarios are looked at.
The contents of this tutorial was presented on the Juju Show (April 10 2019), in abridged form, with a slide deck. PDF available here: Lets-talk-about-credentials.pdf (238.7 KB).
The following prerequisites are assumed as a starting point for this tutorial:
- You’re using Ubuntu 18.04 LTS (Bionic).
- Juju (
v.2.5.2) is installed. See the Installing Juju page.
- You have chosen a backing cloud and have created a controller for it. Refer
to the Clouds page to get started with a cloud and controller.
This guide uses Google GCE as a backing cloud.
This is the list of the discreet operations that are possible when managing credentials:
- Add a local credential
- Remove a local credential
- List local credentials
- List remote credentials
- Set a default local credential
- Relate a remote credential to a model
- Change a local credential
- Change a remote credential
There are all covered on the Credentials page.
Topics covered in this tutorial
- Credential definition
- Local credentials
- Default local credential
- Remote credentials
- Credential used when creating a controller
- Credential used when adding a model
- Tracking what remote credential is used by a model
- Managing credentials with multiple users
- Dealing with expired credentials
- Dealing with a reanimated cloud account
A Juju credential represents a collection of authentication material (like username & password, or client id & secret key) that is specific to a cloud and to a Juju user. This bundle of data is managed by means of a user-defined string: the credential name. This name is not a Juju user nor is it the username of the underlying cloud account. It is an arbitrary label that is assigned when the credential was created with the
Every credential is born (created) from a specific Juju client, which, in turn, is bound to an independent computer host (“device”). There is thus no central way to constrain what credential names get used nor oversee what authentication material they represent. Because of this, different devices can associate the same name with different material, or the opposite, a different name with the same material. This is something to keep in mind, especially in a multi-user context.
When a credential is created it is local to the Juju client that created it. It is therefore known as a local credential and can be listed with the
credentials command. Specifying YAML formatted output makes things explicit:
juju credentials --format yaml
local-credentials: google: gandalf: auth-type: oauth2 client-email: email@example.com client-id: "112674610382273922048" project-id: juju-gce-1725
Notice how the output says ‘local-credentials’.
Here, ‘gandalf’ is the credential name for cloud ‘google’ for the currently logged in Juju user.
Default local credential
When a Juju operation is in need of a local credential it is either specified implicitly, via a defined default local credential, or explicitly, via the
A credential can be defined as the default local credential in two ways:
- Manually by using the
- Automatically when the
bootstrapcommand is invoked
The automatic method is applied when a sole local credential exists when a controller is created.
The default local credential is annotated with an asterisk in the output for the
credentials command. Below, credential ‘tharkun’ is the default local credential for cloud ‘google’ for the currently logged in Juju user:
Cloud Credentials google tharkun*, gandalf, mithrandir
--credential option always refers to a local credential.
In order for Juju to make use of a cloud it must first authenticate with that cloud, and does so via the Juju controller. Therefore, a local credential must first get uploaded from the client to the controller. Once this happens the credential become known as a remote credential. The
show-credentials command lists all remote credentials for the currently logged in Juju user:
controller-credentials: google: gandalf: content: auth-type: oauth2 client-email: firstname.lastname@example.org client-id: "112674610382273922048" project-id: juju-gce-1725 models: controller: admin default: admin
Notice how the output says ‘controller-credentials’.
Here we see that credential ‘gandalf’ is associated with cloud ‘google’ and two models: ‘controller’ and ‘default’. The currently logged in user (the owner of credential ‘gandalf’) is shown to have ‘admin’ model access to both of the aforementioned models.
A remote credential is required whenever a model is created. There are two cases:
- a model is created implicitly via the
- a model is created with the
In the first case a local credential is solicited to become a remote credential and in the second case either a local credential or an existing remote credential is solicited.
Besides creating a new controller, the
bootstrap command also sets up models ‘controller’ and ‘default’.
A remote credential is associated with one cloud, one Juju user, and one or more models.
Note that it is possible for a remote credential to not be related to a model (the model was removed or a private cloud, although very rare, may not require authentication).
Credential used when creating a controller
When a controller is created a local credential is uploaded to the controller and related to models
default. The local credential that gets used is selected according to the following rules, in the order given:
- the one that has been explicitly chosen via the
- the one that has been defined as the default local credential
- the sole existing one
bootstrap command will error out if a credential cannot be found using the above rules. This would occur when the following is true: there are multiple local credentials; the
--credential option was not used; and a default local credential was not set.
Remember: Only local credentials are considered when creating a controller.
Credential used when adding a model
When a model is added a remote credential is related to that model. The remote credential that gets used (for the current Juju user and the given cloud) is selected according to the following rules, in the order given:
- an uploaded local credential, chosen via the
- the sole existing one
Recall that the
--credential option always refers to a local credential. Therefore, in this context, the local credential is first uploaded to the controller, thereby becoming a remote credential, and then related to the model.
add-model command will error out if a credential cannot be found using the above rules. This would occur when the following is true: there are multiple remote credentials and the
--credential option was not used.
Note: There is no concept of an actual default remote credential.
Tracking what remote credential is related to a model
To determine what remote credential is related to a model examine the output of the
show-model command. It will include the model’s credential name, the credential owner (the Juju user who uploaded it), and the cloud:
juju show-model default
default: name: admin/default short-name: default . . . credential: name: jlaurin owner: admin cloud: aws
Above, remote credential ‘jlaurin’ is related to model ‘default’.
As we saw earlier, the
show-credentials command can be used to determine what model (or models) a credential is related to:
controller-credentials: google: saruman: content: . . . models: isengard: admin orthanc: admin
Above, remote credential ‘saruman’ is related to models ‘isengard’ and ‘orthanc’.
If there were multiple remote credentials they would have all been shown. To target a single cloud:credential combination, use the
juju show-credential google saruman
Managing credentials with multiple users
Let’s set up a multi-user context by adding a user and granting that user ‘add-model’ permissions:
juju add-user frodo juju grant frodo add-model
Once this user has registered the controller (
register command) that user may attempt to add a model:
juju add-model shire
This will generate a message indicating that a credential has not been found. This is due to user ‘frodo’ being devoid of both a remote credential and a local credential:
ERROR detecting credentials for "google" cloud provider: gce credentials not found
If multiple local credentials are added, say ‘potatoes’ and ‘beer’, and the operation is re-attempted an error is once again emitted. This time it’s due to not having selected a credential from among the multiple ones available:
ERROR more than one credential is available. List credentials with: juju credentials and then run the add-model command again with the --credential option.
This user can therefore add a model in this way:
juju add-model --credential potatoes shire
Dealing with expired credentials
There have been reports of cloud vendors expiring cloud credentials. The effect of an expired credential is the inability for Juju to administer any models associated with that credential. Note that workloads will continue to run.
To rectify the situation, first contact the cloud vendor and update your existing credential or re-issue a new one. Once that’s done, you will need to do one of the following:
- update the existing Juju credential on the controller
- create a new credential and relate it to affected models
We’ll look at each of these options in greater detail below.
You can simulate an expired credential by deactivating the corresponding account on your cloud’s dashboard or via its API.
Option 1: Update the existing remote credential
To change the remote credential the
update-credential command is used. It does this by uploading the identically named local credential to the controller. Therefore, the local credential first needs to be changed.
The local credential contents are changed with the
add-credential command, where the
--replace option is needed because the credential is already existing.
juju add-credential --replace google -f credentials-gandalf-changed.yaml juju update-credential google gandalf
Here, the contents of credential ‘gandalf’ was modified in file
update-credential command supports the consumption of a YAML file. So here we bypass the local client cache:
juju update-credential google gandalf -f credentials-gandalf-changed.yaml
Option 2: Create a new credential and related it to affected models
To add an entirely new local credential the
add-credential command is used. To relate a remote credential to a model the
set-credential command is used. Here we presume that affected models are ‘shire’ and ‘rohan’:
juju add-credential google -f credentials-mithrandir.yaml juju set-credential -m shire google mithrandir juju set-credential -m rohan google mithrandir
Above, the contents for remote credential ‘mithrandir’ were added to file
A credential targeted with the
set-credential command will upload the identically-named local credential if it is not found remotely.
Dealing with a reanimated cloud account
Closely related to the previous case of credentials expired by the cloud vendor is the scenario where the cloud provider account was deactivated by the cloud account administrator. Every cloud vendor has the functionality to do this and it can be accomplished via the cloud’s API or via its web dashboard.
A deactivated cloud account will naturally also result in an invalid credential on the Juju side. A reasonable way forward would be to simply reactivate the account, but this is insufficient. This is because Juju has already invalidated the credential. The solution is as before: “update” the existing Juju credential on the controller with the
update-credential command. However, you do not need to actually replace the credential’s contents. You just need to announce to the controller that the credential is a good one.
Assuming, therefore, that the account has been made active again and it is associated with Juju credential ‘tharkun’, simply proceed as follows:
juju update-credential google tharkun