Juju bootstrap error from OpenStack

openstack

#1

From the de debug info it looks like API call to Neutron is missing the tenant_id.
But that is in the juju credentials. So what is I missing?
I run a charmed openstack from cs:bundle/openstack-base-61.
juju version 2.6.9-bionic-amd64

I run:
juju bootstrap os-Test os-Test-controller --config network=dcd11aa1-535c-466b-8cae-cb0a96f6ebcd --config external-network=b35dbefa-1b52-4b8d-b042-593b4f8e95ac --config use-floating-ip=true --metadata-source /home/ubuntu/simplestreams

And the error i get is:
Creating Juju controller “os-Test-controller” on os-Test/RegionOne
Looking for packaged Juju agent version 2.6.9 for amd64
Launching controller instance(s) on os-Test/RegionOne…
ERROR failed to bootstrap model: cannot start bootstrap instance: cannot set up groups: failed to create a security group with name: juju-7c4e95df-9ba2-4bc4-80d5-e9cabdfbd51b-7c9a1905-f14d-453a-8a92-589b5fd0ad03
caused by: request (http://10.104.128.139:9696/v2.0/security-groups) returned unexpected status: 400; error info: {“NeutronError”: {“type”: “HTTPBadRequest”, “message”: “Running without keystone AuthN requires that tenant_id is specified”, “detail”: “”}}

This is the credentials
local-credentials:
os-Test:
admin:
auth-type: userpass
domain-name: admin_domain
password: ***************
project-domain-name: admin_domain
tenant-id: a690aecfdeb34c5c8851b24c42e8d7c7
tenant-name: admin
user-domain-name: admin_domain
username: admin

A debug gives this:

12:29:13 INFO  juju.cmd supercommand.go:57 running juju [2.6.9 gc go1.10.4]
12:29:13 DEBUG juju.cmd supercommand.go:58   args: []string{"/snap/juju/9102/bin/juju", "bootstrap", "os-Test", "os-Test-controller", "--config", "network=dcd11aa1-535c-466b-8cae-cb0a96f6ebcd", "--config", "external-network=b35dbefa-1b52-4b8d-b042-593b4f8e95ac", "--config", "use-floating-ip=true", "--metadata-source", "/home/ubuntu/simplestreams", "--debug"}
12:29:13 DEBUG juju.cmd.juju.commands bootstrap.go:1038 authenticating with region "" and credential "admin" ()
12:29:13 DEBUG juju.cmd.juju.commands bootstrap.go:1166 provider attrs: map[use-openstack-gbp:false policy-target-group: use-floating-ip:true use-default-secgroup:false network:dcd11aa1-535c-466b-8cae-cb0a96f6ebcd external-network:b35dbefa-1b52-4b8d-b042-593b4f8e95ac]
12:29:13 INFO  cmd authkeys.go:114 Adding contents of "/home/ubuntu/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
12:29:13 DEBUG juju.cmd.juju.commands bootstrap.go:1225 preparing controller with config: map[max-status-history-size:5G no-proxy:127.0.0.1,localhost,::1 cloudinit-userdata: apt-ftp-proxy: use-default-secgroup:false disable-network-management:false enable-os-upgrade:true automatically-retry-hooks:true logforward-enabled:false network:dcd11aa1-535c-466b-8cae-cb0a96f6ebcd agent-metadata-url: snap-store-assertions: ssl-hostname-verification:true logging-config: firewall-mode:instance apt-no-proxy: juju-no-proxy:127.0.0.1,localhost,::1 snap-https-proxy: juju-https-proxy: container-image-metadata-url: fan-config: resource-tags: backup-dir: http-proxy: net-bond-reconfigure-delay:17 policy-target-group: development:false name:controller apt-mirror: max-action-results-size:5G container-networking-method: image-metadata-url: apt-http-proxy: update-status-hook-interval:5m container-image-stream:released provisioner-harvest-mode:destroyed test-mode:false type:openstack snap-store-proxy-url: proxy-ssh:false https-proxy: apt-https-proxy: max-status-history-age:336h image-stream:released egress-subnets: use-floating-ip:true transmit-vendor-metrics:true ftp-proxy: juju-http-proxy: agent-stream:released authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2R7uDgxrmEGbJyC3OjB4TVdjIr/AR0bmv7D9L+oOF4uTOTfUamtC2j0BZ48LP3o6hFBiX1Ye1S0GpkyHaemo4iA/OTx28dUnT4tUgIYaeOtyGXxt178hR4DC0nSB9ODi8Vot+CnO6GPOoNIDdqU26QyESgzP+i/ILJef4y46ffZf8dXc+oRymsUQL0zgGCHsqWBb8kAzkxrJZKHjbbupENZD7I5bexU8kkeCnRYWt4PkDHRDPT6d1AGpbLIZ/Y1xOs0h0O9YYP11OpFyWqYLelJE5eynoFN5z/vlH8pWpGleEqkez6Zzt4EJj/XL0OTLCAjBGrsH+b8Rdth4lQqyJ juju-client-key
 enable-os-refresh-update:true external-network:b35dbefa-1b52-4b8d-b042-593b4f8e95ac uuid:cdf9fca4-e54d-4b90-8524-327c9f3cf4f2 use-openstack-gbp:false snap-http-proxy: ignore-machine-addresses:false juju-ftp-proxy: container-inherit-properties: snap-store-proxy: default-series:bionic max-action-results-age:336h]
12:29:13 INFO  juju.provider.openstack provider.go:162 opening model "controller"
12:29:14 DEBUG goose client.go:547 auth details: &{Token:gAAAAABdlx8Mz8F4_fmUDfrJgZwSZkPlsJLx0kdJpfhR5g2HLM9QWvaVsQ30xQbK6eTTvXCUaNNY3OL0GMl3aRY5ER3SbRd2RB2lcjOG_Jzy6XfLN3GK_TH4n9BMmFccxGxUzLKtwyGVrz2zR9_g2dMW1gzaej0Vh1tog2bu6Co7N-AW3x1qF9k TenantId: TenantName: UserId:ff7e82e021004290ab8a9988ce6820a9 Domain:admin_domain RegionServiceURLs:map[RegionOne:map[object-store:http://10.104.128.140:80/swift/v1 compute:http://10.104.128.141:8774/v2.1 placement:http://10.104.128.141:8778 network:http://10.104.128.139:9696 identity:http://10.104.128.143:5000/v3 image:http://10.104.128.138:9292]]}
12:29:14 INFO  cmd bootstrap.go:677 Creating Juju controller "os-Test-controller" on os-Test/RegionOne
12:29:14 DEBUG goose apiversion.go:274 discovered API versions: [{Version:{Major:2 Minor:0} Links:[{Href:http://10.104.128.141:8774/v2/ Rel:self}] Status:SUPPORTED} {Version:{Major:2 Minor:1} Links:[{Href:http://10.104.128.141:8774/v2.1/ Rel:self}] Status:CURRENT}]
12:29:15 INFO  juju.cmd.juju.commands bootstrap.go:746 combined bootstrap constraints: 
12:29:15 DEBUG juju.environs.bootstrap bootstrap.go:262 model "controller" supports application/machine networks: true
12:29:15 DEBUG juju.environs.bootstrap bootstrap.go:264 network management by juju enabled: true
12:29:15 DEBUG juju.environs.bootstrap bootstrap.go:884 no agent directory found, using default agent metadata source: https://streams.canonical.com/juju/tools
12:29:15 DEBUG juju.environs.bootstrap bootstrap.go:909 setting default image metadata source: /home/ubuntu/simplestreams/images
12:29:15 DEBUG juju.environs imagemetadata.go:46 new user image datasource registered: bootstrap metadata
12:29:15 INFO  juju.environs.bootstrap bootstrap.go:927 custom image metadata added to search path
12:29:15 INFO  cmd bootstrap.go:296 Loading image metadata
12:29:15 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "bootstrap metadata"
12:29:15 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default cloud images"
12:29:15 DEBUG juju.environs imagemetadata.go:112 obtained image datasource "default ubuntu cloud images"
12:29:15 DEBUG juju.environs.bootstrap bootstrap.go:775 constraints for image metadata lookup &{{{RegionOne http://10.104.128.143:5000/v3} [precise quantal saucy vivid win2012hv win8 xenial win2008r2 win2016 win10 genericlinux bionic win2016nano raring trusty win2012hvr2 disco win2012r2 win2016hv win81 centos7 wily cosmic zesty eoan win2012 win7 utopic yakkety artful opensuseleap] [amd64 i386 armhf arm64 ppc64el s390x] released}}
12:29:15 DEBUG juju.environs.bootstrap bootstrap.go:787 found 1 image metadata in bootstrap metadata
12:29:15 DEBUG juju.environs.bootstrap bootstrap.go:787 found 0 image metadata in default cloud images
12:29:16 DEBUG juju.environs.simplestreams simplestreams.go:457 skipping index "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson" because of missing information: index file has no data for cloud {RegionOne http://10.104.128.143:5000/v3} not found
12:29:16 DEBUG juju.environs.bootstrap bootstrap.go:783 ignoring image metadata in default ubuntu cloud images: index file has no data for cloud {RegionOne http://10.104.128.143:5000/v3} not found
12:29:16 DEBUG juju.environs.bootstrap bootstrap.go:791 found 1 image metadata from all image data sources
12:29:16 INFO  cmd bootstrap.go:373 Looking for packaged Juju agent version 2.6.9 for amd64
12:29:16 INFO  juju.environs.bootstrap tools.go:72 looking for bootstrap agent binaries: version=2.6.9
12:29:16 DEBUG juju.environs.tools tools.go:102 finding agent binaries in stream: "released"
12:29:16 DEBUG juju.environs.tools tools.go:104 reading agent binaries with major.minor version 2.6
12:29:16 DEBUG juju.environs.tools tools.go:112 filtering agent binaries by version: 2.6.9
12:29:16 DEBUG juju.environs.tools tools.go:118 filtering agent binaries by architecture: amd64
12:29:16 DEBUG juju.environs.tools urls.go:116 trying datasource "keystone catalog"
12:29:16 DEBUG juju.environs.simplestreams simplestreams.go:683 using default candidate for content id "com.ubuntu.juju:released:tools" are {20161007 mirrors:1.0 content-download streams/v1/cpc-mirrors.sjson []}
12:29:16 INFO  juju.environs.bootstrap tools.go:74 found 17 packaged agent binaries
12:29:16 INFO  cmd bootstrap.go:467 Starting new instance for initial controller
12:29:17 INFO  cmd bootstrap.go:161 Launching controller instance(s) on os-Test/RegionOne...
12:29:17 DEBUG juju.environs.instances image.go:64 instance constraints {region: RegionOne, series: bionic, arches: [amd64], constraints: mem=3584M, storage: []}
12:29:17 DEBUG juju.environs.instances image.go:70 matching constraints {region: RegionOne, series: bionic, arches: [amd64], constraints: mem=3584M, storage: []} against possible image metadata [{Id:d3e38326-7d1c-4d6b-805e-2e4d54c88fe7 Arch:amd64 VirtType:}]
12:29:17 INFO  juju.environs.instances image.go:106 find instance - using image with id: d3e38326-7d1c-4d6b-805e-2e4d54c88fe7
12:29:17 DEBUG juju.cloudconfig.instancecfg instancecfg.go:956 Setting numa ctl preference to false
12:29:17 DEBUG juju.service discovery.go:64 discovered init system "systemd" from series "bionic"
12:29:17 DEBUG juju.provider.openstack provider.go:1133 openstack user data; 2488 bytes
12:29:17 DEBUG juju.provider.openstack provider.go:1156 using network id "dcd11aa1-535c-466b-8cae-cb0a96f6ebcd"
12:29:17 DEBUG goose apiversion.go:274 discovered API versions: [{Version:{Major:2 Minor:0} Links:[{Href:http://10.104.128.139:9696/v2.0/ Rel:self}] Status:CURRENT}]
12:29:17 ERROR juju.cmd.juju.commands bootstrap.go:697 failed to bootstrap model: cannot start bootstrap instance: cannot set up groups: failed to create a security group with name: juju-42c88823-4335-46ac-8206-87bfb517e02a-cdf9fca4-e54d-4b90-8524-327c9f3cf4f2
caused by: request (http://10.104.128.139:9696/v2.0/security-groups) returned unexpected status: 400; error info: {"NeutronError": {"type": "HTTPBadRequest", "message": "Running without keystone AuthN requires that tenant_id is specified", "detail": ""}}
12:29:17 DEBUG juju.cmd.juju.commands bootstrap.go:698 (error details: [{/build/juju/parts/juju/go/src/github.com/juju/juju/cmd/juju/commands/bootstrap.go:772: failed to bootstrap model} {/build/juju/parts/juju/go/src/github.com/juju/juju/environs/bootstrap/bootstrap.go:562: } {/build/juju/parts/juju/go/src/github.com/juju/juju/environs/bootstrap/bootstrap.go:471: } {/build/juju/parts/juju/go/src/github.com/juju/juju/provider/common/bootstrap.go:56: } {/build/juju/parts/juju/go/src/github.com/juju/juju/provider/common/bootstrap.go:212: cannot start bootstrap instance} {/build/juju/parts/juju/go/src/github.com/juju/juju/provider/openstack/provider.go:1218: } {/build/juju/parts/juju/go/src/github.com/juju/juju/provider/openstack/provider.go:1218: cannot set up groups} {/build/juju/parts/juju/go/src/github.com/juju/juju/provider/openstack/firewaller.go:466: } {failed to create a security group with name: juju-42c88823-4335-46ac-8206-87bfb517e02a-cdf9fca4-e54d-4b90-8524-327c9f3cf4f2
caused by: request (http://10.104.128.139:9696/v2.0/security-groups) returned unexpected status: 400; error info: {"NeutronError": {"type": "HTTPBadRequest", "message": "Running without keystone AuthN requires that tenant_id is specified", "detail": ""}}}])
12:29:17 DEBUG juju.cmd.juju.commands bootstrap.go:1332 cleaning up after failed bootstrap
12:29:17 INFO  juju.provider.common destroy.go:21 destroying model "controller"
12:29:17 INFO  juju.provider.common destroy.go:32 destroying instances
12:29:18 DEBUG juju.provider.openstack provider.go:1509 terminating instances []
12:29:18 INFO  juju.provider.common destroy.go:56 destroying storage
12:29:18 DEBUG juju.provider.openstack cinder.go:646 endpoint "volumev2" not found for "RegionOne" region, trying "volume"
12:29:18 DEBUG juju.provider.openstack cinder.go:646 endpoint "volumev2" not found for "RegionOne" region, trying "volume"
12:29:18 INFO  cmd supercommand.go:502 command finished

#2

Hey @lasse

It may seem counter intuitive but check out mine and @szeestraten responses in this thread regarding the Glance Simple Stream Sync Charm.

I was able to bootstrap my Openstack controller. In my case I did have to use a Disco image for the lxc container hosting the simple streams charm


#3

Thanks @dvnt for leading me in the right direction.

The solution was to remove the ‘domain-name’ field is set in credentials.yaml
See Bug #1806985 “Juju fails to bootstrap on OpenStack with 'domain-...” : Bugs : juju
And thanks for pointing at Glance Simple Stream Sync Charm nice feature to have. Just works on disco for me too.

That’s one small step for mankind, one giant leap for me.