Can't bootstrap internal K8S


#1

Upgraded my juju to 2.6/edge and I get this…

ubuntu@bionic-test:~$ juju add-k8s internal-k8s --region=openstack/RegionOne --storage=openstack-standard

k8s substrate "openstack/RegionOne" added as cloud "internal-k8s" with storage provisioned
by the existing "openstack-standard" storage class.
You can now bootstrap to this cloud by running 'juju bootstrap internal-k8s'.
ubuntu@bionic-test:~$ juju bootstrap internal-k8s
Creating Juju controller "internal-k8s-RegionOne" on internal-k8s/RegionOne
Creating k8s resources for controller "controller-internal-k8s-RegionOne"
ERROR failed to bootstrap model: creating controller stack for controller: creating namespace for controller stack: Namespace "controller-internal-k8s-RegionOne" is invalid: metadata.name: Invalid value: "controller-internal-k8s-RegionOne": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?')

Looks like a bug, but who knows…


#3

Okay that seems to still occur when I switch back to stable channel when bootstrapping…

Can I manually name the namespace? Or is it bust?

I’ve filed https://bugs.launchpad.net/juju/+bug/1842922 because I can’t do any more K8S testing until I figure this out.


#4

hmm, have to look if that’s coming from Juju or Juju proxying that from k8s or what.


#5
ubuntu@bionic-test:~$ kubectl create namespace RicksCoolNamespace

The Namespace "RicksCoolNamespace" is invalid: metadata.name: Invalid value: "RicksCoolNamespace": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?')
ubuntu@bionic-test:~$

Well the error comes from K8S, but a week ago the region wasn’t appended to the controller. Or if it was it wasn’t on my first and only bootstrap.

I mean, the “obvious” fix is to do a .lower() or whatever the Golang equivalent is, but I know nothing about this stuff :slight_smile:


#6

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/229

That seems to say it happens if an executor pod hostname ends up getting truncated ending in “-”

But in this case, it is probably the upper case character since the regex is all lower-case characters.
Juju could certainly Lower() those, but I’am not sure the need/don’t need to add region.


#7

If there’s no distinct requirement for the region, it not being there would be nice as it renders K8S bootstrapping inoperable :slight_smile:


#8

As a possible workaround, could you try creating a name for the controller yourself, e.g. juju bootstrap internal-k8s controller-openstack-k8s?


#9

I’ve filed a bug report in Launchpad for this issue.

https://bugs.launchpad.net/juju/+bug/1842991


#10

@magicaltrout so what’s happening here is as follows.

juju bootstrap takes a controller name as an optional arg; if not supplied it will generate one using <cloudname>-<regionname>.

$ juju bootstrap openstack/RegionOne
-> controller name will be openstack-RegionOne

$ juju bootstrap openstack/RegionOne fred
-> controller name will be fred

For VM clouds, the controller name is just a label.

For k8s clouds, the controller artefacts like the operator and volume claim etc etc all live in a namespace named after the controller: controller-<controllername>.
You can see where this is going - if the controller name contains invalid characters for a namespace name, sad panda.

So the fix right now is to ensure you explicitly name the controller when bootstrapping.
And if there’s already a controller with that same name running in the cluster, Juju will make you choose another name.


#11

Ah yeah

juju bootstrap internal-k8s spicule-k8s

seems happy. Thanks!