Juju agent performance issues

I’m running a model with seven charms deployed, including mongodb, zookeeper, kafka, and two deployments of mariadb. All relations have been made, pods deployed, and the applications should be relatively idle, but I see high cpu usage across the board:

top - 01:03:33 up 13 days, 12:19, 27 users,  load average: 38.16, 39.13, 31.36
Tasks: 913 total,   9 running, 823 sleeping,   0 stopped,   2 zombie
%Cpu(s): 86.4 us, 12.9 sy,  0.0 ni,  0.4 id,  0.1 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 16100884 total,  1059308 free, 10899608 used,  4141968 buff/cache
KiB Swap: 15725396 total, 10249188 free,  5476208 used.  4569876 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND            
 1550 root      20   0 1320996  76724  57612 S  30.6  0.5   4:59.62 jujud
 1243 root      20   0 1394792  77464  57372 S  29.4  0.5   5:12.41 jujud
 1896 root      20   0 1321060  76408  57432 S  29.0  0.5   7:17.95 jujud
 1245 root      20   0 1394856  77464  57016 S  27.1  0.5   7:23.30 jujud
 1344 root      20   0 1394792  79400  57796 S  26.1  0.5   4:57.98 jujud
 1788 root      20   0 1468460  77028  57852 R  23.5  0.5   4:56.76 jujud              
 1614 root      20   0 1394728  75676  57468 S  18.1  0.5   4:59.62 jujud

Looking at juju debug-log, it seems like relation join and change hooks are firing much more frequently than I’d expect. The agent seems to be constantly in an ‘executing’ state. I’ve pastebin’d the logs here: Ubuntu Pastebin

It doesn’t seem to be one particular relation that’s firing repeatedly, but all of them. I see the kafka, mongo, and mysql relations firing frequently, including joins, when the relations are already joined.

Is this a known issue in the current beta (2.5-beta1) or something I should file as a bug?

I’ve not seen it before.
Please do file a bug. If the charms could be uploaded to the staging store and the bug contains the steps to deploy and relate the charms, that would be great so we could reproduce etc

It turns out that this was a bug in the mariadb-k8s charm, where the database.requested state was never cleared, so the hook kept being executed. I’ve confirmed that clearing the flag resolves the performance issue and filed a pull request against the charm with the change.

Thanks Adam! Really appreciate the fix. I just did the charms as a POC to get the Juju k8s stuff working so they do need some love to make them production worthy.

Fix is merged and new charm revision published to staging store.