The deployment of the charm works fine but the worker nodes are showing ‘Waiting for kubelet to start’ at the ‘juju status’ command.
The logs of the ‘snap.kubelet.daemon’ service on the worker node show the error message below.
kubelet.daemon[9522]: F0801 13:07:30.395495 9522 server.go:273] failed to run Kubelet: could not init cloud provider "openstack": Authentication failed
It seems that whatever credentials were passed to Juju are either not getting into the config properly or are missing some sort of permissions to allow Kubernetes to perform the actions it needs.
I would start off by checking the /var/snap/kube-controller-manager/common/cloud-config.conf file to ensure that the credentials there seem reasonable. Beyond that, you would need to check the logs for kube-controller-manager and kubelet to see if you can find some more context about what actions it was attempting when it failed.
Thanks for your reply.
I checked the /var/snap/kube-controller-manager/common/cloud-config.conf file and the credentials there are fine.
Do you know where the logs of kubelet and kube-controller-manager are located?
I just found at the kubernetes-worker for journalctl -u snap.kubelet.daemon.service this output
Aug 14 15:35:51 juju-1eb1bf-default-6 kubelet.daemon[17358]: I0814 15:35:51.513225 17358 controller.go:101] kubelet config controller: starting controller
Aug 14 15:35:51 juju-1eb1bf-default-6 kubelet.daemon[17358]: I0814 15:35:51.513952 17358 controller.go:267] kubelet config controller: ensuring filesystem is set up correctly
Aug 14 15:35:51 juju-1eb1bf-default-6 kubelet.daemon[17358]: I0814 15:35:51.514082 17358 fsstore.go:59] kubelet config controller: initializing config checkpoints directory "/root/cdk/kubelet/dynamic-config/st
Aug 14 15:35:51 juju-1eb1bf-default-6 kubelet.daemon[17358]: I0814 15:35:51.524031 17358 server.go:425] Version: v1.15.2
Aug 14 15:35:51 juju-1eb1bf-default-6 kubelet.daemon[17358]: W0814 15:35:51.524224 17358 plugins.go:118] WARNING: openstack built-in cloud provider is now deprecated. Please use 'external' cloud provider for o
Aug 14 15:35:51 juju-1eb1bf-default-6 kubelet.daemon[17358]: F0814 15:35:51.894530 17358 server.go:273] failed to run Kubelet: could not init cloud provider "openstack": Authentication failed
Aug 14 15:35:51 juju-1eb1bf-default-6 systemd[1]: snap.kubelet.daemon.service: Main process exited, code=exited, status=255/n/a
Aug 14 15:35:51 juju-1eb1bf-default-6 systemd[1]: snap.kubelet.daemon.service: Failed with result 'exit-code'.
Aug 14 15:35:52 juju-1eb1bf-default-6 systemd[1]: snap.kubelet.daemon.service: Service hold-off time over, scheduling restart.
Aug 14 15:35:52 juju-1eb1bf-default-6 systemd[1]: snap.kubelet.daemon.service: Scheduled restart job, restart counter is at 5.
Aug 14 15:35:52 juju-1eb1bf-default-6 systemd[1]: Stopped Service for snap application kubelet.daemon.
Do you know where the logs of kubelet and kube-controller-manager are located?
I just found at the kubernetes-worker for journalctl -u snap.kubelet.daemon.service
That is the correct log for kubelet, and the appropriate one for kube-controller-manager would be on kubernetes-master and viewable with:
journalctl -u snap.kube-controller-manager.daemon
I checked the /var/snap/kube-controller-manager/common/cloud-config.conf file and the credentials there are fine.
If kubelet is getting an authentication error but the credentials in the config file match what you granted to the integrator, then it would seem that the issue is how those credentials are set up in OpenStack. I would recommend trying those credentials with the OpenStack CLI client (I find snap install openstackclients to be easy to install and use) and ensure that you can connect to the cloud properly with that. If so, then perhaps you could post a redacted copy of the credentials as they appear in the cloud-config.conf file so that we can confirm if they are formatted correctly, etc.
