My 2 cents -
The public clouds provide apt caches local to the az. To verify this, deploy something to gce or aws and cat your apt sources.list.
MAAS can be configured to run a local apt-cache which you can configure juju to use.
This leaves a few cloud types without native apt-cache options; openstack, lxd and manual.
I agree with @timClicks reasoning, but I feel maintaining the apt-cache on the controller and as a part of juju could be more of a PIA then its worth. Some of my apt-caches grow to 100+GB and experience heavy iops and network load (depending). I would hate to see the resource usage of the apt-cache interfere with the operational capability of the juju controller in any way. For this reason alone, I don’t feel that running an apt-cache on the controller is a good idea.
As far as speeding up the lxd bootstrap, possibly we could provide a pre-baked lxd juju-controller image instead of downloading/installing it all at bootstrap.