CMaRrrrgh!

juju
1

Building a CMR across two controllers. All units, including controllers exist in the same switch and subnet. Network interfaces are bridged on physical machines, no MACVLAN.
Here’s how it looks right now:

qzaMdYm.png1920×636 108 KB

The steps, after the ceph units were deployed are:
$ juju offer ceph-mon:mon --controller lxdcluster-default
$ juju add-relation ceph-osd lxdcluster-default:admin/ceph.ceph-mon --model maas:admin/basecloud
This command errors that no relations were found. However the SAAS appears with active status, as visible in the screenshot above. If I attempt to remove the SAAS it errors that the remote application doesn’t exist.

2

Well, i tore it all down and built again, and it’s still broken, but in a different way that perhaps is more traceable. I create my offer, then when instead of going straight for add-relation I do
$ juju consume lxdcluster-default:admin/ceph.ceph-mon mon
The SAAS appears, this time with status ‘unknown’. I don’t see any reference to it in debug-log on either end of the pipe, is there a better way to debug CMR?

On repeated tests, it seems random if the SAAS appears with status ‘active’ or status ‘unknown’.

3

Does anyone have any pointers on debugging CMR offers? I’ve got so many windows open it’s getting chilly in here!

  • 2 controller debug-logs
  • 2 model debug-logs
  • 2 status watches
  • 2 for terminals
  • 1 for buying another monitor from Amazon

I don’t seem to get any debug output related to the relations at all. If I call the commands with juju --debug I do see the API connections but it all seems fine. (I am assuming here that ‘RPC Connection died’ is a standard output when an API operation completes, as I see that a lot).

4

On the debug-log…

by default, once a controller has come up, the log level is set to WARNING. So debug messages will not appear unless turned on.
You can activate DEBUG level logging for just the cmr bits:

juju model-config -m controller logging-config="<root>=WARNING;unit=DEBUG;juju.apiserver.common.crossmodel=DEBUG;juju.apiserver.crossmodelrelations=DEBUG;juju.worker.remoterelations=DEBUG"
1 Like
5

The wrong endpoint has been offered.
What’s been offered is the “mon” endpoint which is for peer relations between the ceph-mon units themselves.

The charm metadata defines these “provides” endpoints:

provides:
  nrpe-external-master:
    interface: nrpe-external-master
    scope: container
  client:
    interface: ceph-client
  osd:
    interface: ceph-osd
  radosgw:
    interface: ceph-radosgw
  bootstrap-source:
    interface: ceph-bootstrap

You want to offer the ceph-osd endpoint.

Cross model relations are no different to non-cross model relations here. There can only be a relation between endpoints offering the same interface protocol, where one end is “provides” and the other “requires”. You would have got the same “no relations found” error even in a single model scenario.

1 Like
6

With the “known” vs “active” status of the SAAS application…

The consuming side watches the offering side to report the status of the offered workload. I have seen the odd scenario where this doesn’t always get reported. The root cause hasn’t been found yet. Juju 2.5 beta2 does make some improvements around cross controller comms but I suspect there still may be a latent issue there. It’s annoying but doesn’t affect the operation of the relation.

1 Like