It appears that the index file is missing images for the LON region.
02:04:35 DEBUG juju.environs.simplestreams simplestreams.go:457 skipping index http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson" because of missing information: index file has no data for cloud {LON https://lon.identity.api.rackspacecloud.com/v2.0} not found
juju --version
2.5.2-bionic-amd64
This makes Juju unusable in RackSpace London.
That certainly does appear to be the case.
We’ll check with our people who publish the image metadata to see what may have happened and try to get it fixed ASAP.
I am considering using API to export the needed image(s) from one of the other regions, importing it to LON region as public, shared image(s), getting IDs of the image(s) and using local simplestreams on bootstrap.
What’s your take @wallyworld? Plausible?
Plausible yes. I am yet to hear back what may have happened to the London image metadata. Hopefully we’ll know more soon.
While waiting to hear back from people who publish the image metadata create a local image metadata source and bootstrap. 
Follow Cloud image metadata how-to, to set up a local image metadata source.
Replace the content of simplestreams/images/streams/v1/index.json with the one given below.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
{
"index": {
"com.ubuntu.cloud:released:rax": {
"updated": "Sat, 13 Apr 2019 09:53:48 +0200",
"format": "products:1.0",
"datatype": "image-ids",
"cloudname": "rax",
"clouds": [
{
"region": "LON",
"endpoint": "https://lon.identity.api.rackspacecloud.com/v2.0/"
}
],
"path": "streams/v1/com.ubuntu.cloud:released:rax.json",
"products": [
"com.ubuntu.cloud:server:18.04:amd64"
]
}
},
"updated": "Sat, 13 Apr 2019 09:53:48 +0200",
"format": "index:1.0"
}
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJcsdOmAAoJEIwCuqMngwalAtYH/iibgK2hb8YryqJlzSUfSXK8
NyhRakFZ+ikwSNtmNW+3MoxkNxmdBLmfsw38Pd5v4GPGYkR+he3Pl2KoRiV9+BTA
Rt9lopi+vzEtNq8LWLNy3SAZMJcfGCkkMPz7OMruBwNqwimB4UyZbJ00aG3/v5IS
cHxag+vi4D6ZfLHcZjY6uM4y2kQtNa0oCIQIxTi1l7z2q/X0EAHBIyKbHCUIKBFR
IWaqKboN6O0WqUo7VcPb9mkBPJsrEqIh0Az8mT+zjv1l8g/OvMRM580mIUVjTuca
Ef9XQB2lAHDxfXDvP6gi+49ZacRsTJHQE5lx959UtrGnUx7qu27sDdpoc+Y8/2w=
=WaW4
-----END PGP SIGNATURE-----
Create file simplestreams/images/streams/v1/com.ubuntu.cloud:released:rax.json having the content given below.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
{
"updated": "Sat, 13 Apr 2019 09:53:48 +0200",
"content_id": "com.ubuntu.cloud:released:rax",
"products": {
"com.ubuntu.cloud:server:18.04:amd64": {
"release": "bionic",
"version": "18.04",
"arch": "amd64",
"versions": {
"(LATEST)": {
"items": {
"LON-pvhvm": {
"virt": "pvhvm",
"pubname": "Ubuntu 18.04 LTS (Bionic Beaver) (PVHVM)",
"crsn": "LON",
"id": "6e731e8e-ff3d-4e27-be6c-147fde688b7c",
"endpoint": "https://lon.identity.api.rackspacecloud.com/v2.0/"
}
}
}
}
}
},
"format": "products:1.0",
"_aliases": {
"crsn": {
"LON": {
"region": "LON",
"endpoint": "https://lon.identity.api.rackspacecloud.com/v2.0/"
}
}
}
}
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJcsdTmAAoJEIwCuqMngwal/28H/i5lAs6Tiuquozs+2bHvDDuq
QMLOY0b7ZAuPFuTNMI7YSmJP0a9NtJXvXzvjfLS9VPZ19XZxYgKKb1AHDmqz7n6p
Q5C43oNN/TGL32Rhg8lIv6g3jFDsEcMxQ8F5XeQev3xkGXnKDGbdeKqeMYYGaGeN
f75G5bbmezr0e64fWBOThcFw08ni0i44EKyYDoaXSW1pNWe48Ft926fHLAcTwJ9I
qp1iQ6FfxqD0xMJ/sLbhH93YeQaEALKgpJ9QApuWZVADXczXnjimpsN3SA+kQ4uC
k9vjYUG0lpkThBoFPIMFms37OtWWo64K4dPQIdP/t1jEZaziY8NwtG3f5c6al/E=
=xnEr
-----END PGP SIGNATURE-----
To bootstrap in the London region controller named controller-rax-lon run
juju bootstrap rackspace/lon controller-rax-lon --metadata-source ~/simplestreams
Publicly available Ubuntu 18.04 LTS will be used to create the controller.
To get the list of the all available Ubuntu images in the RackSpace cloud London region, authenticate to RackSpace and run
curl -i -X GET \
-H "Content-Type:application/json" \
-H "X-Auth-Token:<your token here>" \
'https://lon.images.api.rackspacecloud.com/v2/images?visibility=public&os_distro=ubuntu'
Add other images to com.ubuntu.cloud:released:rax.json as needed.
Bootstrap Juju controller in RackSpace Cloud London region using local image metadata
Good news…
I have been informed that the issue has been resolved and the region is now appearing in the RAX streams.
http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:rax.json
Hi there,
So after rackspace to remove the OS image that schkovich used when bootstrapping our controller using simplestream. Wallyworld, drove through enabling the default image-metadata feature which is usually used to list and manage OS images.
That permit to list (juju metadata list-images) and delete the removed image that can not be used while using juju add-machine or juju add-unit.
Then due to image-metadata being enabled, I could then, add a new image with an image ID that rackspace told me to use.
Here now are the issues.
juju metadata add-image does not stay longer than about 5 minutes in the list, it just disappear.image-metadata breaks other scripts (python jujulib) with something like “Unknown facade” or sojuju remove-machine otherwise the list of instance just grows (I stopped after around 10)… After a long time, all instances appear ready in rackspace, but of course not manageable with juju
I tried this on two different models.
How would juju add-unit would create several instances with the same name?
It does seem like the image metadata for Rackspace is empty on cloud-images.ubuntu.com, so using custom image metadata is currently required.
Ideally, when needing to use custom image metadata, the --metadata-source bootstrap parameter is used when setting up the controller - in that case, the metadata records would not expire. But in this case, the controller has already been bootstrapped. It may be easier to bootstrap another controller with the correct image metadata and migrate models to that new controller.
For an already bootstrapped controller, any custom image metadata records added after bootstrap are set to expire after 5 minutes (for historical reasons) - the current implementation expects that the model config image-metadata-url is preferred. So if it is not desired to bootstrap a new controller and migrate models across, you’ll need to set up a small http server pointing to custom image metadata and set image-metadata-url accordingly. Bootstrapping a new controller and migrating would definitely be easier I think.
Since Rackspace is Openstack based, it may be possible to follow these instructions to add image metadata to keystone
The Python libjuju issue is because this ability to add custom image metadata via the Juju CLI is experimental (hence the use of a feature flag) and is not necessarily supported by all add on libraries.
With the machine start issue, it seems Rackspace could be sending back an unexpected status code that Juju doesn’t interpret correctly. The Rackspace support in Juju is built as a thin wrapper around the Openstack provider. There’s been many times subtle differences / non-compliant API implementations and/or cloud config has caused Juju to have issues. Once way to see what might be happening is to look at the Juju log files after enabling debugging
juju model-config logging-config="<root>=INFO;unit=INFO;juju.provider.openstack=DEBUG"
One workaround might be to provision the machine manually and then use juju add-machine to add that instance to the Juju model. You can then deploy to that machine using deploy --to <N>. It’s far from perfect but may help in the interim.
Hi WallyWorld,
Thanks for your reply.
I have two questions.
image-metadata-url, what would it has to server exactly? a json file directly? what is the expected content of that file?Regards,
David
Yes, you can migrate a model to a new controller which has been set up to use custom image metadata.
The http server just has to server static content (which you can edit by hand as needed). Here’s the inbuilt image metadata URL used for the published pubic cloud image metadata
Under that URL at path /streams/v1/ is the metadata - a single json image file which points to product files, one for each cloud. You can see the content of the Rackspace data in the rax files.
Your image-metadata-url needs serve data which contains a similar directory structure.
Hi Wally,
I do first bootstrap a new controller with simplestream details with those following parameters:
juju bootstrap rackspace/lon <ctrl-name> --metadata-source ~/simplestreams --config network=XXXXX-cc70-43bf-a8ec-489be09855cb --config use-floating-ip=true --bootstrap-constraints "cores=2 mem=4G root-disk=32G"
I seems to start great but then fails with a 403, I suspect something missing on rackspace side:
Creating Juju controller "tom-rax-lon" on rackspace/lon Looking for packaged Juju agent version 2.7.6 for amd64 Launching controller instance(s) on rackspace/lon... ERROR failed to bootstrap model: cannot start bootstrap instance: cannot run instance: failed to run a server with nova.RunServerOpts{Name:"juju-5b892b-controller-0", FlavorId:"5", ImageId:"498bfa38-d288-4e37-b975-a4acb77f8557", UserData:[]uint8{0x1f, 0x8b, 0x8.... ... 0x0}, SecurityGroupNames:[]nova.SecurityGroupName{}, Networks:[]nova.ServerNetworks{nova.ServerNetworks{NetworkId:"00000000-0000-0000-0000-000000000000", FixedIp:"", PortId:""}, nova.ServerNetworks{NetworkId:"11111111-1111-1111-1111-111111111111", FixedIp:"", PortId:""}, nova.ServerNetworks{NetworkId:"XXXXXXXX-cc70-43bf-a8ec-489be09855cb", FixedIp:"", PortId:""}}, AvailabilityZone:"", Metadata:map[string]string{"juju-model-uuid":"0fce44a9-7684-40a6-87a2-7290415b892b", "juju-controller-uuid":"6dac6a84-e7a5-4902-8882-474cb09f9ac6", "juju-is-controller":"true"}, ConfigDrive:true, BlockDeviceMappings:[]nova.BlockDeviceMapping{nova.BlockDeviceMapping{BootIndex:0, UUID:"498bfa38-d288-4e37-b975-a4acb77f8557", SourceType:"image", DestinationType:"local", VolumeSize:0, VolumeType:"", DeleteOnTermination:true, DeviceName:"", DeviceType:"", DiskBus:"", GuestFormat:"", NoDevice:false, Tag:""}}} caused by: Unauthorised URL https://lon.servers.api.rackspacecloud.com/v2/<tenantId>/servers caused by: request (https://lon.servers.api.rackspacecloud.com/v2/<tenantID>/servers) returned unexpected status: 403; error info: {"forbidden": {"message": "Policy doesn't allow standard_flavor:create:attach_volume to be performed.", "code": 403}}
Any idea what could that be?
Thanks
You don’t seem to be able to attach volumes.
It seems like a Rackspace setup issue doesn’t it? I’m guessing there’s policy setting you can change for the tenant you’re logging in as to enable volumes to be attached to the types of image your are requesting. I don’t have specific Rackspace knowledge to tell you what button to click sorry. Perhaps another @juju-developers person can chime in?
Thanks for your feedback,
User has all permissions regarding servers managements…
There is no fine grained permission (like until attaching volumes)
Hi Wally,
Yeah sounds like a setup changed between “before” and “nowadays”. Of course I will say that nothing within the rax account has been changed for the last 6 month , and last time we did manage (re creating containers etc…) was in February.
Now creating a new controller seems to undercover some deeper changes in rax infrastructure.
Would it be any other option, like to use the current controller but to tell him to use this “new” imageID?
Even if I think if that would be possible, I am not sure that I would have the same kind of error anyway.
But if there is I would like to try that.
RAX asked me to change the flavor to use to create the bootstrap instance.
using instance-type I would have a list of possible flavors, unfortunately each name contains spaces like 4GB Standard Instance which end up not making juju client happy
valid values are: [1 GB Classic v1 1 GB General Purpose v1 1 GB Performance 120 GB I/O v1 120 GB Memory v1 120 GB Performance 15 GB Classic v1 15 GB Compute v1 15 GB I/O v1 15 GB Memory v1 15 GB Performance 15GB Standard Instance 1GB Standard Instance 2 GB Classic v1 2 GB General Purpose v1 2 GB Performance 240 GB Memory v1 256 MB Classic v1 2GB Standard Instance 3.75 GB Compute v1 30 GB Classic v1 30 GB Compute v1 30 GB I/O v1 30 GB Memory v1 30 GB Performance 30GB Standard Instance 4 GB Classic v1 4 GB General Purpose v1 4 GB Performance 4GB Standard Instance 512 MB Classic v1 512MB Standard Instance 60 GB Compute v1 60 GB I/O v1 60 GB Memory v1 60 GB Performance 7.5 GB Compute v1 8 GB Classic v1 8 GB General Purpose v1 8 GB Performance 8GB Standard Instance 90 GB I/O v1 90 GB Performance OnMetal General Purpose v2 Large OnMetal General Purpose v2 Medium OnMetal General Purpose v2 Small OnMetal I/O v2] davidt@srv-wds01:~$ juju bootstrap rackspace/lon tom-rax-lon --metadata-source ~/simplestreams --config network=dd3389c6-cc70-43bf-a8ec-489be09855cb --bootstrap-constraints 'instance-type=4GB Standard Instance' ERROR malformed constraint "Standard"
Any advice?
Hi there (again),
After several trial to create a new controller it is still failing.
The last command line used is juju bootstrap rackspace/lon tom-rax-lon --metadata-source ~/simplestreams --config network=dd3389c6-cc70-43bf-a8ec-489be09855cb --bootstrap-constraints "cores=4 mem=4G"
Which does not finish properly.
The last answer from rackspace support is this:
All of our current images are PVHVM. Are you still using image id ‘498bfa38-d288-4e37-b975-a4acb77f8557’ for your builds? This image is PVHVM, but regardless the issue we are seeing shouldn’t be related to the virtualization style of the server. The real issue comes down to the bootstrapping process since the nova-agent is not being installed during the bootstrapping process Rackspace does. The bootstrapping process requires that the virtual machine set its IP via cloud-init and then reach out to https://mirror.rackspace.com/ospc/bootstrap to installed xenserver client utilities and nova-agent. If this process fails then your server will hang at the 80%/90% mark as the openstack code is such that it is waiting for a response from the nova-agent service in your server.
the image ID the quote contains is the one another support member told me to use, earlier (about a month ago by now) and which is apparently fine (as the container is creating fine)
The issue seems to be more about nova-agent not being able to install properly or so, but I don’t have much visibility in that regard.
Any help appreciated
Regards,
David