Changing vSphere credentials, requires Juju controller restart?

We are testing out vSphere as a cloud for Juju, with very good results, and will soon use this for production purposes. We found a possible issue with changing cloud credentials though.

The vCenter Server uses Active Directory for authentication, and due to a policy passwords need to be changed in AD every now and then. This causes the controller to lose contact with vCenter, and in our case one authentication error per second in the vCenter admin console.

I had some trouble solving this at first and wanted to share the experiences to see if I’m doing it right.

In this example I will use the cloud esxiuba and the credential johan.

Credentials available locally in my Juju client:

hallback@t1000:~$ juju list-credentials
Cloud           Credentials
esxiuba         johan*, johan-new

credentials.yaml:

credentials:
  esxiuba:
    default-credential: johan
    johan:
      auth-type: userpass
      password: PWVISIBLEINPLAINTEXT
      user: johan@my.company.com
    johan-new:
      auth-type: userpass
      password: PWVISIBLEINPLAINTEXT
      user: johan

Credentials available remotely on Juju controller in vSphere:

hallback@t1000:~$ juju show-credentials
controller-credentials:
  esxiuba:
    johan:
      content:
        auth-type: userpass
        user: johan@my.company.com
      models:
        controller: admin
        erikp-model: admin
    johan-new:
      content:
        auth-type: userpass
        user: johan
      models: {}

I experimented with switching to another valid account while troubleshooting. At first i tried to add a second credential (hence “johan-new”), but failed. Instead, I tried later to replace the old one locally on my client:

hallback@t1000:~$ juju add-credential --replace esxiuba
Enter credential name: johan

A credential "johan" already exists locally on this client.
Replace local credential? (y/N): y

Using auth-type "userpass".

Enter user: erik@my.company.com

Enter password: 

Credential "johan" updated locally for cloud "esxiuba".

Now I replaced the credentials on the controller:

hallback@t1000:~$ juju update-credential esxiuba johan     
Updated credential "johan" for user "admin" on cloud "esxiuba".

In order to see if it worked, I observed in vCenter which user actually deploys the VMs. The credential should have been changed to erik@my.company.com. What I observed:

  • Adding a new unit still uses the old account/credential
  • Deploying a new charm in the same model still uses the old account/credential
  • Adding a new model and deploying a charm does use the NEW account/credential
  • Switching back to the original model and deploying again uses the OLD account/credential

The only way I found so far to get rid of this was to restart services on the Juju controller (juju-db.service might have been unneccessary). Is this right or am I doing it wrong?

# sudo systemctl stop jujud-machine-0.service
# sudo systemctl stop juju-db.service
# sudo systemctl start juju-db.service
# sudo systemctl start jujud-machine-0.service
1 Like

Can you please file this as a bug under bugs.launchpad.net/juju ?

I think we agree that things shouldn’t need a restart after an update-credential command.

2 Likes

I agree with rick_h - it should not happen. In fact, the cloud credentials that a model uses are just references to the cloud credentials on the controller. I’ll dig deeper into bug once you file it and I am guessing that you were using Juju 2.5? Please confirm on the bug.

It might be helpful for other users and future references to have the bug linked to this post.

Sorry for the late reply, but just wanted to update and bring some clarity. I was using Juju 2.4.7, and i tried today with 2.5.1. It works now! Several improvements have obviously been fixed since 2.4.7.

If I change my password used for vSphere, this is all that needs to be done now:

$ juju add-credential --replace esxiuba
Enter credential name: johan

A credential "johan" already exists locally on this client.
Replace local credential? (y/N): y

Using auth-type "userpass".

Enter user: username@my.company.com

Enter password: 

Credential "johan" updated locally for cloud "esxiuba".


$ juju update-credential esxiuba johan
Credential valid for:
  default
  controller
Controller credential "johan" for user "admin" on cloud "esxiuba" updated.
For more information, see ‘juju show-credential esxiuba johan’.

No restarts required.

Another thing that also seems to be working better now is that once the controller realizes the credentials are invalid, it stops trying to login. In 2.4.7, the controller was logging in and failing almost every second, driving our vSphere admins insane. This only appeared once in the log:

e634027f-22b4-400f-8fca-d4f57fb8238a: machine-0 2019-02-15 13:31:40 ERROR juju.worker.dependency engine.go:636 "compute-provisioner" manifold worker returned unexpected error: failed to process updated machines: failed to get all instances from broker: dialing client: ServerFaultCode: Cannot complete login due to an incorrect user name or password.

Great!

2 Likes

This is great to hear \o/ Thank you for the feedback and the update!!

1 Like