Juju agent performance issues


#1

I’m running a model with seven charms deployed, including mongodb, zookeeper, kafka, and two deployments of mariadb. All relations have been made, pods deployed, and the applications should be relatively idle, but I see high cpu usage across the board:

top - 01:03:33 up 13 days, 12:19, 27 users,  load average: 38.16, 39.13, 31.36
Tasks: 913 total,   9 running, 823 sleeping,   0 stopped,   2 zombie
%Cpu(s): 86.4 us, 12.9 sy,  0.0 ni,  0.4 id,  0.1 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 16100884 total,  1059308 free, 10899608 used,  4141968 buff/cache
KiB Swap: 15725396 total, 10249188 free,  5476208 used.  4569876 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND            
 1550 root      20   0 1320996  76724  57612 S  30.6  0.5   4:59.62 jujud
 1243 root      20   0 1394792  77464  57372 S  29.4  0.5   5:12.41 jujud
 1896 root      20   0 1321060  76408  57432 S  29.0  0.5   7:17.95 jujud
 1245 root      20   0 1394856  77464  57016 S  27.1  0.5   7:23.30 jujud
 1344 root      20   0 1394792  79400  57796 S  26.1  0.5   4:57.98 jujud
 1788 root      20   0 1468460  77028  57852 R  23.5  0.5   4:56.76 jujud              
 1614 root      20   0 1394728  75676  57468 S  18.1  0.5   4:59.62 jujud

Looking at juju debug-log, it seems like relation join and change hooks are firing much more frequently than I’d expect. The agent seems to be constantly in an ‘executing’ state. I’ve pastebin’d the logs here: http://paste.ubuntu.com/p/6nXGxtwmTK/

It doesn’t seem to be one particular relation that’s firing repeatedly, but all of them. I see the kafka, mongo, and mysql relations firing frequently, including joins, when the relations are already joined.

Is this a known issue in the current beta (2.5-beta1) or something I should file as a bug?


#2

I’ve not seen it before.
Please do file a bug. If the charms could be uploaded to the staging store and the bug contains the steps to deploy and relate the charms, that would be great so we could reproduce etc


#3

It turns out that this was a bug in the mariadb-k8s charm, where the database.requested state was never cleared, so the hook kept being executed. I’ve confirmed that clearing the flag resolves the performance issue and filed a pull request against the charm with the change.


#4

Thanks Adam! Really appreciate the fix. I just did the charms as a POC to get the Juju k8s stuff working so they do need some love to make them production worthy.

Fix is merged and new charm revision published to staging store.