Changing vSphere credentials, requires Juju controller restart?


#1

We are testing out vSphere as a cloud for Juju, with very good results, and will soon use this for production purposes. We found a possible issue with changing cloud credentials though.

The vCenter Server uses Active Directory for authentication, and due to a policy passwords need to be changed in AD every now and then. This causes the controller to lose contact with vCenter, and in our case one authentication error per second in the vCenter admin console.

I had some trouble solving this at first and wanted to share the experiences to see if I’m doing it right.

In this example I will use the cloud esxiuba and the credential johan.

Credentials available locally in my Juju client:

hallback@t1000:~$ juju list-credentials
Cloud           Credentials
esxiuba         johan*, johan-new

credentials.yaml:

credentials:
  esxiuba:
    default-credential: johan
    johan:
      auth-type: userpass
      password: PWVISIBLEINPLAINTEXT
      user: johan@my.company.com
    johan-new:
      auth-type: userpass
      password: PWVISIBLEINPLAINTEXT
      user: johan

Credentials available remotely on Juju controller in vSphere:

hallback@t1000:~$ juju show-credentials
controller-credentials:
  esxiuba:
    johan:
      content:
        auth-type: userpass
        user: johan@my.company.com
      models:
        controller: admin
        erikp-model: admin
    johan-new:
      content:
        auth-type: userpass
        user: johan
      models: {}

I experimented with switching to another valid account while troubleshooting. At first i tried to add a second credential (hence “johan-new”), but failed. Instead, I tried later to replace the old one locally on my client:

hallback@t1000:~$ juju add-credential --replace esxiuba
Enter credential name: johan

A credential "johan" already exists locally on this client.
Replace local credential? (y/N): y

Using auth-type "userpass".

Enter user: erik@my.company.com

Enter password: 

Credential "johan" updated locally for cloud "esxiuba".

Now I replaced the credentials on the controller:

hallback@t1000:~$ juju update-credential esxiuba johan     
Updated credential "johan" for user "admin" on cloud "esxiuba".

In order to see if it worked, I observed in vCenter which user actually deploys the VMs. The credential should have been changed to erik@my.company.com. What I observed:

  • Adding a new unit still uses the old account/credential
  • Deploying a new charm in the same model still uses the old account/credential
  • Adding a new model and deploying a charm does use the NEW account/credential
  • Switching back to the original model and deploying again uses the OLD account/credential

The only way I found so far to get rid of this was to restart services on the Juju controller (juju-db.service might have been unneccessary). Is this right or am I doing it wrong?

# sudo systemctl stop jujud-machine-0.service
# sudo systemctl stop juju-db.service
# sudo systemctl start juju-db.service
# sudo systemctl start jujud-machine-0.service

#2

Can you please file this as a bug under bugs.launchpad.net/juju ?

I think we agree that things shouldn’t need a restart after an update-credential command.


#3

I agree with rick_h - it should not happen. In fact, the cloud credentials that a model uses are just references to the cloud credentials on the controller. I’ll dig deeper into bug once you file it and I am guessing that you were using Juju 2.5? Please confirm on the bug.

It might be helpful for other users and future references to have the bug linked to this post.