Part of https://discourse.jujucharms.com/t/read-before-contributing/47, an opinionated guide by William Reade
See go doc ./worker/catacomb
and go doc ./worker/dependency
for
details; the TLDRs are roughly:
-
dependency.Engine
allows you to run a network of
interdependent tasks, each of which is represented by a
dependency.Manifold
which knows how to run the task, and what
resources are needed to do so. - Tasks are started in essentially random order, and restarted when
any of their dependencies either starts or stops; or when they
themselves stop. This converges pretty quickly towards a stable set
of workers running happily; and (sometimes) a few that are
consistently failing, and all of whose dependents are dormant
(waiting to be started with an available dependency). - Two tasks that need to share information in some way should
generally not depend on one another: they should share a
dependency on a resource that represents the channel of
communication between the two. (The direction of information flow is
independent of the direction of dependency flow, if you like.) - However, you can simplify workers that depend on mutable
configuration, by making them a depend upon a resource that
supplies that information to clients, but also watches for changes,
and bounces itself when it sees a material difference from its
initial state (thus triggering dependent bounces and automatic
reconfiguration with the fresh value). Seeworker/lifeflag
and
worker/migrationflag
for examples; and seeworker/environ
for
the Tracker implementation (mentioned above) which takes advantage
ofenvirons.Environ
being goroutine-safe to share a single value
between clients and update it in the background, thus avoiding
bounces. - You might want to run your own
dependency.Engine
, but you’re
rather more likely to need to add a task to theManifolds
func in
the relevant subpackages ofcmd/jujud/agent
(depending on what
agent the task needs to run in).
…and:
-
catacomb.Catacomb
allows you to robustly manage the lifetime of a
worker.Worker
and any number of additional non-shared Workers. - See the boilerplate in the worker.Worker section, or the docs, for
how to invoke it. - To use it effectively, remember that it’s all about responsibility
transfer.Add
takes unconditional responsibility for a supplied
worker: if the catacomb isKill
ed, so will be that worker; and if
worker stops with an error, the catacomb will itself beKill
ed. - This means that worker can register private resources and forget
about them, rather than having to worry about their lifetimes; and
conversely it means that those resources need implement only the
worker interface, and can avoid having to leak lifetime information
via inappropriate channels (literally).
Between them, they seem to cover most of the tricky situations that come
up when considering responsibility transfer for workers; and since you
can represent just about any time-bounded resource as a worker, they
make for a generally useful system for robustly managing resources that
exist in memory, at least.
All Our Manifolds Are In The Wrong Place
…because they’re in worker packages, alongside the workers, and thus
severely pollute the context-independence of the workers, which can and
should stand alone.
The precise purpose of a manifold is to encapsulate a worker for use in
a specific context: one of the various agent dependency engines. It’s
at the manifold level that we define the input resources, and at the
manifold level that we (should) filter worker-specific errors and render
them in a form appropriate to the context.
(For example, some workers sometimes return dependency.ErrMissing
or
dependency.ErrUninstall
– this is, clearly, a leak of engine-specific
concerns into the wrong context. The worker should return, say,
local.ErrCannotRun
: and the manifold’s filter should convert that
appropriately, because it’s only at that level that it makes sense to
specify the appropriate response. The worker really shouldn’t know it’s
running in a dependency.Engine
at all.)
Next time someone has a moment while doing agent work, they should just
dump all the manifold implementations in appropriate subpackages of
./cmd/jujud/agent
and see where that takes us. Will almost certainly
be progress…