Unable to remove relations, apps or units


#1

I am finding that I am unable to remove anything less than a whole model. This is a consistent problem that keeps coming up.

If I try to remove something using the GUI, and commit, then everything looks like it went through correctly, but the graph shows that the thing I tried to remove is still there.

Similarly with the juju command. juju remove-relation and juju remove-application both appear to work. (Return value 0, nothing printed), but nothing actually happens.

opentelco@dell02:~$ juju remove-relation neutron-api mysql --debug
23:50:53 INFO  juju.cmd supercommand.go:57 running juju [2.5.4 gc go1.10.4]
23:50:53 DEBUG juju.cmd supercommand.go:58   args: []string{"/snap/juju/7206/bin/juju", "remove-relation", "neutron-api", "mysql", "--debug"}
23:50:53 INFO  juju.juju api.go:67 connecting to API addresses: [10.58.2.36:17070]
23:50:53 DEBUG juju.api apiclient.go:888 successfully dialed "wss://10.58.2.36:17070/model/5cb0e921-1540-4b05-81d2-bf1f5e39ac4c/api"
23:50:53 INFO  juju.api apiclient.go:608 connection established to "wss://10.58.2.36:17070/model/5cb0e921-1540-4b05-81d2-bf1f5e39ac4c/api"
23:50:53 DEBUG juju.api monitor.go:35 RPC connection died
23:50:53 INFO  cmd supercommand.go:502 command finished
opentelco@dell02:~$ juju add-relation neutron-api mysql
cannot add relation "neutron-api:shared-db mysql:shared-db": relation neutron-api:shared-db mysql:shared-db already exists (already exists)

Is RPC connection died a bad thing? It’s a DEBUG message, so I am assuming that is expected behavior.

I can delete a whole model, then rebuild. When it try to remove a relation/app within the new model, the same thing happens.

juju --version shows 2.5.4-bionic-amd64. My controller and workload are deployed in MAAS.


#2

What does juju status show after you remove a relation?

Juju entities generally go through a lifecycle, and the asynchronous nature of Juju means that actions don’t always happen immediately.

Removing a relation requires hooks to run on the respective units before the relation is fully removed.

The relation will move from alive -> dying -> dead -> removed.


#3

It shows the same thing as before.

juju status does not show relations. The units are the same. Typically they were already in an error state, and I was trying to fix them. But not always.


#4

The output to juju status --relations will display the relations.

Reference: https://docs.jujucharms.com/charms-relations


#5

Ok I’m trying again (on a different unit).

I’m looking at juju status --relations. As I expected, that just shows nothing has changed.

I have a unit in an error state, because the hook when a relationship to a database changes is a broken hook. (Either I have configured the charm wrong, or the hook has a bug in it.)
If I try to delete that relationship, Juju runs the hook, which fails, so the relation never gets deleted.

How am I supposed to delete it? I feel like I am drastically misunderstanding something basic about Juju, because it just seems like it is rarely possible to remove or modify anything after something is deployed.

The broken unit is neutron-api. The relevant lines from juju status --relations are:

neutron-api             12.0.5         error        1  neutron-api             jujucharms  269  ubuntu
neutron-api/0*             error     idle   3        10.58.1.215     9696/tcp                    hook failed: "shared-db-relation-changed" for mysql:shared-db
...
mysql:shared-db                            neutron-api:shared-db                       mysql-shared                    regular

I ran

juju remove-relation neutron-api mysql

Looking at juju status --relations I see that nothing has changed. No logs appear in juju debug-log --include neutron-api/0 --level WARN

So then I run juju resolved neutron-api/0 repeatedly until Juju says there is nothing to resolve.

juju status --relations shows that the neutron-api/0 unit has changed to now have comment Incomplete relations: database, identity, and status executing, for a few seconds. (The relationship is still there.) Then after a few seconds it just goes back to how it was originally. And juju debug-log has a new entry saying

unit-neutron-api-0: 06:22:08 ERROR juju.worker.uniter.operation hook "shared-db-relation-changed" failed: exit status 1

So how can I remove a relationship if one of the hooks called during removal fails?