K8S Stuck deployment

juju
1

I have a test K8S charm that deployed then I tried add-unit and it added a new pod thats up and running, but Juju says hook failed: start and resolved does similar weird broken stuff…

application-saiku-k8s: 23:10:34 INFO unit.saiku-k8s/2.juju-log status-set: maintenance: Configuring Saiku container
application-saiku-k8s: 23:10:34 INFO unit.saiku-k8s/2.juju-log set pod spec:
containers:
  - name: saiku-k8s
    imageDetails:
      imagePath: registry.jujucharms.com/spiculecharms/saiku-k8s/saiku_image@sha256:1c1cbf89281237ec287deb14b42b48b985384b50c25e5f335934f52b2f8f108b
      username: docker-registry
      password: <pass>
    ports:
    - containerPort: 8080
      protocol: TCP


application-saiku-k8s: 23:10:35 ERROR juju.worker.uniter.context saiku-k8s/2 is not the leader but is setting application pod spec
application-saiku-k8s: 23:10:35 ERROR unit.saiku-k8s/2.juju-log pod-spec-set encountered an error: `ERROR this unit is not the leader`
application-saiku-k8s: 23:10:35 ERROR unit.saiku-k8s/2.juju-log Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-saiku-k8s-2/charm/reactive/saiku_k8s.py", line 34, in config_mariadb
    layer.caas_base.pod_spec_set(spec)
  File "lib/charms/layer/caas_base.py", line 16, in pod_spec_set
    run(['pod-spec-set'], stdout=PIPE, stderr=PIPE, check=True, input=spec.encode('utf-8'))
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.

application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed Traceback (most recent call last):
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed   File "/var/lib/juju/agents/unit-saiku-k8s-2/charm/hooks/leader-settings-changed", line 19, in <module>
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed     main()
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed   File "lib/charms/reactive/__init__.py", line 74, in main
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed     bus.dispatch(restricted=restricted_mode)
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed   File "lib/charms/reactive/bus.py", line 390, in dispatch
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed     _invoke(other_handlers)
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed   File "lib/charms/reactive/bus.py", line 359, in _invoke
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed     handler.invoke()
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed   File "lib/charms/reactive/bus.py", line 181, in invoke
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed     self._action(*args)
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed   File "/var/lib/juju/agents/unit-saiku-k8s-2/charm/reactive/saiku_k8s.py", line 34, in config_mariadb
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed     layer.caas_base.pod_spec_set(spec)
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed   File "lib/charms/layer/caas_base.py", line 16, in pod_spec_set
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed     run(['pod-spec-set'], stdout=PIPE, stderr=PIPE, check=True, input=spec.encode('utf-8'))
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed   File "/usr/lib/python3.6/subprocess.py", line 438, in run
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed     output=stdout, stderr=stderr)
application-saiku-k8s: 23:10:35 DEBUG unit.saiku-k8s/2.leader-settings-changed subprocess.CalledProcessError: Command '['pod-spec-set']' returned non-zero exit status 1.
application-saiku-k8s: 23:10:35 ERROR juju.worker.uniter.operation hook "leader-settings-changed" failed: exit status 1
application-saiku-k8s: 23:10:55 INFO unit.saiku-k8s/2.juju-log Reactive main running for hook leader-settings-changed

The

application-saiku-k8s: 23:08:46 ERROR juju.worker.uniter.context saiku-k8s/2 is not the leader but is setting application pod spec
application-saiku-k8s: 23:08:46 ERROR unit.saiku-k8s/2.juju-log pod-spec-set encountered an error: `ERROR this unit is not the leader`

lines seem pretty sad.

Any ideas? Bug? User error?

2

It seems on the surface this could be an issue with the charm.

Only the leader can set the pod spec. Usually, unit 0 would be the leader and the fact that unit 2 seems to be setting the pod spec is therefore an issue. There’s a charm helpers is_leader() method that can be used to see if a unit is the leader or not.

Note also that the leader-settings-changed hook is never run on the lead unit, so it’s not surprising that there’s a “not the leader” error trying to do a leader only operation.

juju status will put an asterisk next to the leader unit so you can check that as well to see what the lead unit is.