Help unpicking my MAAS Mess


#1

Hey folks!

TL;DR: I think I’ve got a race condition where unitdata is being populated before the data is made available by the service starting. What’s the right way to prevent this? I’ve tried to set flags once the service is started, and i’m looking for some function to test for the availability of a file.

A series of unfortunate broken external dependencies led me to hack @jamesbeedy’s excellent MAAS charm, so it can install via apt rather than snap. Of course I made rather a mess of this, being this is my early forays into these depths of implementation, and I’m overstepping my rather limited python boundaries by a significant margin.

I branched my fork to separate the layer into 2 charms, for rack and region, as I thought this may make things easier to debug. They’re here (rack) and here (region).

The deploy itself goes like this:

  1. Deploy Postgres to new model
  2. Deploy layer-maas-region-lxd
  3. Relate Postgres to Maas-region
  4. Deploy layer-maas-rack-lxd
  5. Relate region to rack
  6. ???
  7. Profit!

Right now it’s failing at the end of step 3, as the Charm attempts to add the MAAS secret to unitdata.kv. It’s looking for a file that MAAS creates, and it’s not finding it. If I look for the file in the unit (/var/lib/maas/secret) it’s there, but I assume it’s either not there when the hook for unitdata.kv is triggered, or there’s some permissions issue?

I’ve had a go using debug-hooks, chlp, charms.reactive.sh and manual hook triggering, but I feel a bit lost to be honest. I am however not looking for someone to fix this for me, I’d much rather figure it out myself - but after several days poking at this it feels like I need to gain some outside perspective. I would like to document this process as an attempt to give insight into debugging and probably some other best practices I need to adopt, but I’m not there yet.


#2

“looking for a file that MAAS creates”.

Can you clarify which charm is failing? Is it MAAS-region that is looking for the file, or Postgres?

If MAAS-region is looking for a file that MAAS creates, do you have an idea when it is created? Is it created in response to the relation? Or is it just done as part of some stage of initialization, and the charm hook is simply racing with MAAS finishing the setup of the machine?

Assuming it is the charm hook running on the machine, it is unlikely to be a permission issue, as hooks default to running as root. (I doubt the charm hook would be calling setuid to drop power.)

I have the feeling that the “look for the file” needs to be changed to some sort of poll / look for some other event that signals the file should be there by now.