Hi all
I’m trying to wrap my head around how units and relations work with k8s charms. I have a setup of two charms connected to each other: sse-endpoint-mock and sse-consumer.
It seems as if every time my charm executes pod_spec_set
, a new juju unit agent gets created and the old one shuts down. Each agent seems to have its own reactive state, flags and hook lifecycle.
The initial relation and configuration works fine. Stuff gets weird however when I change a config option on the endpoint and it propagates from the endpoint to the consumer resulting in a redeploy of the pod in k8s.
This is what happens at the consumer side when it receives a new base-url
config over its relation.
-
Hook
sse-endpoint-relation-changed
runs. It sets a new pod spec with the updatedbase-url
. I get a warning that the charm isn’t the leader, even though only one unit is present (see 15:18:52). Is this a bug? -
Suddenly, a new agent appears (con6/2 -> con6/3). Does updating a pod spec create a new Juju agent? This is really confusing, can you explain this more?
-
The start hook gets executed as the new unit. This adds to the confusion because this can kickstart a new deployment etc… So it’s possible for a running pod to exist when an agent initially boots? Is the reactive state shared between agents? Is the relationship shared between agents? Does the charm at the other side now see two agents? How do you know whether you’re the agent creating the pod or the agent created by creating the pod?
-
leader-settings-changed
runs as the new unit. More confusion since it’s the same operator pod that becomes the new leader? Does this have anything to do with the issue in #1? -
config-changed
runs. Oh well, I’m used toconfig-changed
running at the strangest times. This isn’t an issue when you use the reactiveconfig.changed
flags since they only signal real config changes. -
sse-endpoint-relation-departed
runs… Now this took me by complete surprise. Why is there a relationship departing? Apparently the old unit is shutting down now?
# 1. Hook `sse-endpoint-relation-changed` runs and sets a new pod spec .
#
application-con6: 15:18:50 INFO unit.con6/2.juju-log sse-endpoint:8: Reactive main running for hook sse-endpoint-relation-changed
application-con6: 15:18:51 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer>
application-con6: 15:18:51 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer: hooks phase, 0 handlers queued
application-con6: 15:18:51 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer>
tracer: ++ queue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:51 INFO unit.con6/2.juju-log sse-endpoint:8: Invoking reactive handler: ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:51 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer: set flag sse-endpoint.changed
application-con6: 15:18:51 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer>
application-con6: 15:18:51 INFO unit.con6/2.juju-log sse-endpoint:8: Invoking reactive handler: reactive/sse-consumer.py:35:config_consumer
application-con6: 15:18:51 INFO unit.con6/2.juju-log sse-endpoint:8: status-set: maintenance: Configuring consumer container
application-con6: 15:18:52 INFO unit.con6/2.juju-log sse-endpoint:8: set pod spec:
application-con6: 15:18:52 WARNING juju.worker.uniter.context con6/2 is not the leader but is setting application pod spec
application-con6: 15:18:52 INFO unit.con6/2.juju-log sse-endpoint:8: status-set: maintenance: creating container
application-con6: 15:18:52 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer>
application-con6: 15:18:52 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer: cleared flag sse-endpoint.changed
application-con6: 15:18:52 INFO unit.con6/2.juju-log sse-endpoint:8: status-set: active: Pods created ()
application-con6: 15:18:52 DEBUG unit.con6/2.sse-endpoint-relation-changed lib/charms/layer/status.py:140: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
application-con6: 15:18:52 DEBUG unit.con6/2.sse-endpoint-relation-changed includes = yaml.load(fp.read()).get('includes', [])
application-con6: 15:18:52 INFO juju.worker.uniter.operation ran "sse-endpoint-relation-changed" hook
application-con6: 15:18:53 INFO juju.worker.leadership con6 leadership for con6/3 denied
application-con6: 15:18:53 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-con6-3
application-con6: 15:18:53 INFO juju.worker.uniter unit "con6/3" started
application-con6: 15:18:53 INFO juju.worker.uniter hooks are retried true
application-con6: 15:18:53 INFO juju.worker.uniter found queued "start" hook
# 2. A new unit agent appears?
# 3. Hook `start` runs on new agent.
application-con6: 15:18:54 INFO unit.con6/3.juju-log Reactive main running for hook start
application-con6: 15:18:54 DEBUG unit.con6/3.juju-log tracer>
tracer: ++ queue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:14:broken:sse-endpoint
tracer: ++ queue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:54 DEBUG unit.con6/3.juju-log tracer>
application-con6: 15:18:54 DEBUG unit.con6/3.juju-log tracer: hooks phase, 0 handlers queued
application-con6: 15:18:54 DEBUG unit.con6/3.juju-log tracer>
tracer: ++ queue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:14:broken:sse-endpoint
tracer: ++ queue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:54 INFO unit.con6/3.juju-log Invoking reactive handler: reactive/sse-consumer.py:24:notify_relation_needed
application-con6: 15:18:54 INFO unit.con6/3.juju-log Invoking reactive handler: ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:14:broken:sse-endpoint
application-con6: 15:18:54 DEBUG unit.con6/3.juju-log tracer>
tracer: -- dequeue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:14:broken:sse-endpoint
application-con6: 15:18:54 INFO unit.con6/3.juju-log Invoking reactive handler: ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:55 INFO unit.con6/3.juju-log status-set: blocked: Please add relationship to sse endpoint.
application-con6: 15:18:55 DEBUG unit.con6/3.start lib/charms/layer/status.py:140: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
application-con6: 15:18:55 DEBUG unit.con6/3.start includes = yaml.load(fp.read()).get('includes', [])
application-con6: 15:18:55 INFO juju.worker.uniter.operation ran "start" hook
# 4. leader-settings-changed on new agent
#
application-con6: 15:18:55 INFO unit.con6/3.juju-log Reactive main running for hook leader-settings-changed
application-con6: 15:18:55 DEBUG unit.con6/3.juju-log tracer>
application-con6: 15:18:56 DEBUG unit.con6/3.juju-log tracer: hooks phase, 0 handlers queued
application-con6: 15:18:56 DEBUG unit.con6/3.juju-log tracer>
tracer: ++ queue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:56 INFO unit.con6/3.juju-log Invoking reactive handler: reactive/sse-consumer.py:24:notify_relation_needed
application-con6: 15:18:56 INFO unit.con6/3.juju-log Invoking reactive handler: ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:56 INFO unit.con6/3.juju-log status-set: blocked: Please add relationship to sse endpoint.
application-con6: 15:18:56 DEBUG unit.con6/3.leader-settings-changed lib/charms/layer/status.py:140: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
application-con6: 15:18:56 DEBUG unit.con6/3.leader-settings-changed includes = yaml.load(fp.read()).get('includes', [])
application-con6: 15:18:56 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook
# 5. config-changed on new agent
#
application-con6: 15:18:57 INFO unit.con6/3.juju-log Reactive main running for hook config-changed
application-con6: 15:18:57 DEBUG unit.con6/3.juju-log tracer>
application-con6: 15:18:57 DEBUG unit.con6/3.juju-log tracer: hooks phase, 0 handlers queued
application-con6: 15:18:57 DEBUG unit.con6/3.juju-log tracer>
tracer: ++ queue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:57 INFO unit.con6/3.juju-log Invoking reactive handler: reactive/sse-consumer.py:24:notify_relation_needed
application-con6: 15:18:57 INFO unit.con6/3.juju-log Invoking reactive handler: ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:57 INFO unit.con6/3.juju-log status-set: blocked: Please add relationship to sse endpoint.
application-con6: 15:18:58 DEBUG unit.con6/3.config-changed lib/charms/layer/status.py:140: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
application-con6: 15:18:58 DEBUG unit.con6/3.config-changed includes = yaml.load(fp.read()).get('includes', [])
application-con6: 15:18:58 INFO juju.worker.uniter.operation ran "config-changed" hook
# 6. Hook `sse-endpoint-relation-departed` runs on OLD agent
#
application-con6: 15:18:58 INFO unit.con6/2.juju-log sse-endpoint:8: Reactive main running for hook sse-endpoint-relation-departed
application-con6: 15:18:58 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer>
tracer: ++ queue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:59 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer>
application-con6: 15:18:59 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer: hooks phase, 0 handlers queued
application-con6: 15:18:59 DEBUG unit.con6/2.juju-log sse-endpoint:8: tracer>
tracer: ++ queue handler ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:59 INFO unit.con6/2.juju-log sse-endpoint:8: Invoking reactive handler: reactive/sse-consumer.py:24:notify_relation_needed
application-con6: 15:18:59 INFO unit.con6/2.juju-log sse-endpoint:8: Invoking reactive handler: ../../application-con6/charm/hooks/relations/sse-endpoint/requires.py:20:base_url_changed:sse-endpoint
application-con6: 15:18:59 INFO unit.con6/2.juju-log sse-endpoint:8: status-set: blocked: Please add relationship to sse endpoint.
application-con6: 15:18:59 DEBUG unit.con6/2.sse-endpoint-relation-departed lib/charms/layer/status.py:140: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
application-con6: 15:18:59 DEBUG unit.con6/2.sse-endpoint-relation-departed includes = yaml.load(fp.read()).get('includes', [])
application-con6: 15:18:59 INFO juju.worker.uniter.operation ran "sse-endpoint-relation-departed" hook
I tried to debug this further but since juju debug-log
filtering doesn’t seem to be working, it’s really hard as I have to grep my way through the logs, but many log entries are multiline…
So the tl;dr of all my questions is:
Can you please explain how creating a pod spec influences the Juju agents, the reactive framework state, the operator pods, relations and hooks?