Configuring SSL for open-stack environment, setting certs in juju fails on clients

I’ve been only partially successfully securing an OpenStack environment with SSL certs provided by a CA.
I’ve enabled DNS HA with maas for proper hostname resolution and have a CA wild card for the domain. I set FQDN for each host post-deployment(I have questions about this later) on maas. From what I can tell the configurations are not being set on the deployed applications(charms?) I’ve attempted several different SSL “strings”, a bundle base 64 encoded(issued cert, intermediate cert, root) as well as a straight string of the cert. I’ve noticed that some applications want pem, some want base64 encoded and it looks like others want raw text copy of the cert. I’m looking for some assistance determining the correct way of implementing ssl for OpenStack(Placement, cinder, glance, keystone, neutron-api, rabbitmq-server, ceph-radosgw, nova-cloud-controller, OpenStack-dashboard), I’m having the most difficulty with rabbitmq, every time I attempt SSL configuration the charm fails with errors.

=SUPERVISOR REPORT==== 23-Dec-2019::02:53:37 ===
     Supervisor: {<0.315.0>,tcp_listener_sup}
     Context:    start_error
     Reason:     {shutdown,
                     {failed_to_start_child,ranch_acceptors_sup,
                         {listen_error,
                             {acceptor,{0,0,0,0,0,0,0,0},5671},
                             {options,{cacertfile,[]}}}}}
     Offender:   [{pid,undefined},
                  {id,{ranch_listener_sup,{acceptor,{0,0,0,0,0,0,0,0},5671}}},
                  {mfargs,
                      {ranch_listener_sup,start_link,
                          [{acceptor,{0,0,0,0,0,0,0,0},5671},
                           1,ranch_ssl,
                           [{port,5671},
                            {ip,{0,0,0,0,0,0,0,0}},
                            {max_connections,infinity},
                            {ack_timeout,5000},
                            {connection_type,supervisor},
                            inet6,
                            {backlog,128},
                            {nodelay,true},
                            {linger,{true,0}},
                            {exit_on_close,false},
                            {versions,['tlsv1.2','tlsv1.1',tlsv1]},
                            {verify,verify_peer},
                            {fail_if_no_peer_cert,false},
                            {certfile,"/etc/rabbitmq/rabbit-server-cert.pem"},
                            {keyfile,
                                "/etc/rabbitmq/rabbit-server-privkey.pem"}],
                           rabbit_connection_sup,[]]}},
                  {restart_type,permanent},
                  {shutdown,infinity},
                  {child_type,supervisor}]


=CRASH REPORT==== 23-Dec-2019::02:53:37 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.196.0>
    registered_name: []
    exception exit: {bad_return,
                     {{rabbit,start,[normal,[]]},
                      {'EXIT',
                       {{case_clause,
                         {error,
                          {{shutdown,
                            {failed_to_start_child,
                             {ranch_listener_sup,
                              {acceptor,{0,0,0,0,0,0,0,0},5671}},
                             {shutdown,
                              {failed_to_start_child,ranch_acceptors_sup,
                               {listen_error,
                                {acceptor,{0,0,0,0,0,0,0,0},5671},
                                {options,{cacertfile,[]}}}}}}},
                           {child,undefined,
                            'rabbit_tcp_listener_sup_:::5671',
                            {tcp_listener_sup,start_link,
                             [{0,0,0,0,0,0,0,0},
                              5671,ranch_ssl,
                              [inet6,
                               {backlog,128},
                               {nodelay,true},
                               {linger,{true,0}},
                               {exit_on_close,false},
                               {versions,['tlsv1.2','tlsv1.1',tlsv1]},
                               {verify,verify_peer},
                               {fail_if_no_peer_cert,false},
                               {certfile,
                                "/etc/rabbitmq/rabbit-server-cert.pem"},
                               {keyfile,
                                "/etc/rabbitmq/rabbit-server-privkey.pem"}],
                              rabbit_connection_sup,[],
                              {rabbit_networking,tcp_listener_started,
                               ['amqp/ssl',
                                [{backlog,128},
                                 {nodelay,true},
                                 {linger,{true,0}},
                                 {exit_on_close,false},
                                 {versions,['tlsv1.2','tlsv1.1',tlsv1]},
                                 {verify,verify_peer},
                                 {fail_if_no_peer_cert,false},
                                 {certfile,
                                  "/etc/rabbitmq/rabbit-server-cert.pem"},
                                 {keyfile,
                                  "/etc/rabbitmq/rabbit-server-privkey.pem"}]]},
                              {rabbit_networking,tcp_listener_stopped,
                               ['amqp/ssl',
                                [{backlog,128},
                                 {nodelay,true},
                                 {linger,{true,0}},
                                 {exit_on_close,false},
                                 {versions,['tlsv1.2','tlsv1.1',tlsv1]},
                                 {verify,verify_peer},
                                 {fail_if_no_peer_cert,false},
                                 {certfile,
                                  "/etc/rabbitmq/rabbit-server-cert.pem"},
                                 {keyfile,
                                  "/etc/rabbitmq/rabbit-server-privkey.pem"}]]},
                              1,"SSL Listener"]},
                            transient,infinity,supervisor,
                            [tcp_listener_sup]}}}},
                        [{rabbit_networking,start_listener0,5,
                          [{file,"src/rabbit_networking.erl"},{line,225}]},
                         {rabbit_networking,'-start_listener/5-lc$^0/1-0-',5,
                          [{file,"src/rabbit_networking.erl"},{line,213}]},
                         {rabbit_networking,start_listener,5,
                          [{file,"src/rabbit_networking.erl"},{line,213}]},
                         {rabbit_networking,'-boot_ssl/1-lc$^0/1-0-',3,
                          [{file,"src/rabbit_networking.erl"},{line,136}]},
                         {rabbit_networking,boot_ssl,1,
                          [{file,"src/rabbit_networking.erl"},{line,136}]},
                         {rabbit_networking,boot,0,
                          [{file,"src/rabbit_networking.erl"},{line,122}]},
                         {rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,
                          [{file,"src/rabbit_boot_steps.erl"},{line,49}]},
                         {rabbit_boot_steps,run_step,2,
                          [{file,"src/rabbit_boot_steps.erl"},{line,49}]}]}}}}
      in function  application_master:init/4 (application_master.erl, line 134)
    ancestors: [<0.195.0>]
    message_queue_len: 1
    messages: [{'EXIT',<0.197.0>,normal}]
    links: [<0.195.0>,<0.33.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 157
  neighbours:

As far as I can tell the error is with the SSL certs I’m trying to use. I have tried: raw txt from the cert, base64 encoded version of the same, configuring a bundle(need details as some apps want the bundle setup one way verses another, ie no ca just intermediate and client vs client and intermediate “order matters”)

Can I get some guidance on SSL configurations supplied via JUJU for OpenStack deployments?

I did some digging and it looks like JUJU is not configuring the appropriate files on rabbitMQ.
I configured rabbitmq-server with no certs and enabled SSL and SSHed to the system and as expected found contents in the following files:
/etc/rabbitmq/rabbit-server-ca.pem
/etc/rabbitmq/rabbit-server-cert.pem
/etc/rabbitmq/rabbit-server-privkey.pem

I attempted to reconfigure rabbit-server by supplying my certs (raw text and base64 encoded). I SSHed back to the system and found the contents as follows:
/etc/rabbitmq/rabbit-server-ca.pem empty, expected
/etc/rabbitmq/rabbit-server-cert.pem, unchanged
/etc/rabbitmq/rabbit-server-privkey.pem, empty not expected

I’m not sure what do to at this time.

Sorry for the issues @nathan-flowers, we’re all coming back from holiday break and I’ll see if some folks with more experience can get in and help with setting up the rabbitmq-server with custom certs.

@beisner @fnordahl please send halp! Thanks!

So I did some more testing:
I attempted to supply certs in the following format:
-----BEGIN CERTIFICATE-----
Cert txt here
-----END CERTIFICATE-----

and

cat issued.crt intermediate.crt Root.crt | base64

then coppied the contents in to JUJU gui for the config string.
both options result in “Hook failed” errors.

The files:
/etc/rabbitmq/rabbit-server-ca.pem
/etc/rabbitmq/rabbit-server-cert.pem
/etc/rabbitmq/rabbit-server-privkey.pem
are empty when supplying inforation via JUJU gui.

Thanks for the feedback.

So what are your findings when applying the certificates through the juju CLI?

This appears to be similar to this thread, and if you are seeing the same it may suggest an issue isolated to the Juju GUI.

Sorry for the delay, as far as I can tell Juju gui does work. I can deploy certs to the dashboard and it will work, it’s only rabbitMQ that seams to not work. I didn’t know about vault, i’ll be looking into that for cert management. For reference is there information about using already signed certs with vault?
I don’t want to have to waste the money I already spent on my current cert.

Ive moved on to using vault for SSL management.
I attempted to follow the following webpage without success.
https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-certificate-management.html#upload-signed-csr-and-root-ca-cert-to-vault

i was able to setup vault according to this page:
https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-vault.html

When I run juju run-action vault/0 --wait get-csr country=AU province=State organization=XYZ.wigets
I get this error:

The problem I have right now is when I attempt to upload a signed CSR i get the following error: message: ‘hvac.exceptions.InternalServerError: stored CA information not able to
be parsed’

When ever i do attempt to upload the signed CSR I get this error:
juju run-action vault/0 --wait upload-signed-csr pem="$(cat vault.pem | base64)" root-ca="$(cat cacert.pem | base64)" allowed-domains=‘openstack.local’
unit-vault-0:
UnitId: vault/0
id: “16”
message: ‘hvac.exceptions.InvalidRequest: the given certificate is not marked for
CA use and cannot be used with this backend’
results:
Stderr: |
/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault_pki.py:202: DeprecationWarning: Call to deprecated function ‘_post’. This method will be removed in version ‘0.8.0’ Please use the ‘post’ method on the ‘hvac.adapters’ class moving forward.
json={‘certificate’: pem})
status: failed
timing:
completed: 2020-02-09 22:03:58 +0000 UTC
enqueued: 2020-02-09 22:03:54 +0000 UTC
started: 2020-02-09 22:03:57 +0000 UTC


can anyone offer guidance?

Hey nathan-flowers,

Were you able to find a fix for the error geneated while uploading a signed CSR to vault ?

Thanks!

No, unfortunately i have not found any solution to this issue.
I decided to make Vault the CA, that works at least.

Thank you nathan-flowers for the update. In my case instead of setting vault as the CA, I bought a wildcard cert and used it for all the openstack services. I used the same for vault too.

I was wanting to go this route but didn’t want to purchase the cert until I could verify proof of concept working, which I could never accomplish.