Installing Charmed OSM on AWS


#1

This guide will walk you through installing the Charmed Distribution of OSM on AWS.

Requirements

AWS account

Getting started

Install the basic prerequisites.

sudo snap install juju --classic
sudo snap install kubectl --classic

AWS credentials

Add the credentials of your AWS account.

juju add-credential aws

You will be asked for the auth-type (choose access-key), access-key, and secret-key.

Enter credential name: osm-ha-credential
Using auth-type "access-key".
Enter access-key: XXXXXXXXXXXXXXXXX
Enter secret-key:
Credential "osm-ha-credential" added locally for cloud "aws".

Bootstrap Juju on AWS

Bootstrap the Juju controller, on AWS, which will be used to deploy Charmed Kubernetes.

juju bootstrap aws aws-osm-ha --credential osm-ha-credential

Deploy OSM VCA

The VNF Configuration and Abstraction (VCA) is a component required by Life Cycle Management (LCM), which is another OSM component. The VCA is an LXD bootstraped Juju controller that OSM will use to deploy proxy charms.

juju add-model vca
juju deploy cs:~charmed-osm/vca \
            --constraints "mem=4G cores=2 root-disk=40G" \
            osm-vca
juju offer osm-vca:osm-vca

Deploy LMA stack (optional)

Create graylog-vhost.tmpl

<Location "/">
    RequestHeader set X-Graylog-Server-URL "http://{{ servername }}/api/"
    ProxyPass http://{{ graylog_web }}/
    ProxyPassReverse http://{{ graylog_web }}/
</Location>

<Location "/api/">
    ProxyPass http://{{ graylog_api }}/api/
    ProxyPassReverse http://{{ graylog_api }}/api/
</Location>
Create osm-lma-bundle.yml
series: bionic
variables:
  nagios-context: &nagios-context osm-lma
applications:
  apache2:
    charm: cs:apache2
    constraints: cores=1 mem=4G
    num_units: 1
    expose: true
    options:
      enable_modules: "headers proxy_html proxy_http"
      vhost_http_template: include-base64://graylog-vhost.tmpl
  elasticsearch:
    charm: cs:elasticsearch
    num_units: 1
    constraints: cores=8 mem=32G
    options:
      firewall_enabled: False
      es-heap-size: 16
  graylog:
    charm: cs:graylog
    num_units: 1
    constraints: cores=4 mem=8G
  mongodb:
    charm: cs:mongodb
    constraints: cores=2 mem=4G
    num_units: 1
  grafana:
    charm: cs:grafana
    constraints: cores=2 mem=4G
    num_units: 1
  prometheus:
    charm: cs:prometheus2
    constraints: cores=2 mem=4G
    num_units: 1
  nagios:
    num_units: 1
    constraints: cores=2 mem=4G
    charm: cs:nagios
  nrpe-host:
    charm: cs:nrpe
    options:
      nagios_hostname_type: "host"
      nagios_host_context: *nagios-context
      xfs_errors: "30"
      swap: ''
  filebeat:
    charm: cs:filebeat
    options:
      logpath: "/var/log/*.log /var/log/*/*.log /var/log/syslog"
  telegraf:
    charm: cs:telegraf
relations:
  - [ "apache2:reverseproxy", "graylog:website" ]
  - [ "graylog:elasticsearch", "elasticsearch:client" ]
  - [ "graylog:mongodb", "mongodb:database" ]
  - [ "filebeat:logstash", "graylog:beats" ]
  - [ "prometheus:grafana-source", "grafana:grafana-source" ]
  - [ "telegraf:prometheus-client", "prometheus:target" ]
  - [ "apache2:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "elasticsearch:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "graylog:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "mongodb:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "grafana:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "prometheus:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "nagios:monitors", "nrpe-host:monitors" ]
  - [ "nagios:juju-info", "telegraf:juju-info" ]
  - [ "nagios:juju-info", "filebeat:beats-host" ]

Deploy the LMA stack

juju add-model lma
juju deploy osm-lma-bundle.yml lma-stack
juju offer lma.graylog:beats graylog-beats
juju offer lma.prometheus:target prometheus-target
juju offer lma.nagios:monitors nagios-monitors

Deploy Charmed Kubernetes (without LMA)

Charmed Kubernetes provides a well integrated, turn-key Kubernetes® platform that is open, extensible, and secure.

Create overlay.yaml

applications:
  kubernetes-worker:
    num_units: 3
    constraints: cores=4 mem=8G root-disk=100G
  aws-integrator:
    charm: cs:~containers/aws-integrator
    num_units: 1
relations:
  - ['aws-integrator', 'kubernetes-master']
  - ['aws-integrator', 'kubernetes-worker']

Deploy Charmed Kubernetes

juju add-model cdk
juju deploy charmed-kubernetes --overlay overlay.yaml
juju trust aws-integrator

Deploy Charmed Kubernetes (with LMA)

Charmed Kubernetes provides a well integrated, turn-key Kubernetes® platform that is open, extensible, and secure.

Create overlay.yaml

variables:
  nagios-context: &nagios-context osm-cdk
applications:
  aws-integrator:
    charm: cs:~containers/aws-integrator
    num_units: 1
  kubernetes-master:
    options:
      nagios_context: *nagios-context
  kubernetes-worker:
    num_units: 3
    constraints: cores=4 mem=8G root-disk=100G
    options:
      nagios_context: *nagios-context
  filebeat:
    charm: cs:filebeat
    options:
      logpath: '/var/log/*.log /var/log/*/*.log /var/log/syslog'
  filebeat-kube:
    charm: cs:filebeat
    options:
      logpath: '/var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/pods/*/*/*.log'
      kube_logs: true
  telegraf:
    charm: cs:telegraf
  nrpe-host:
    charm: cs:nrpe
    options:
      nagios_hostname_type: host
      nagios_host_context: *nagios-context
      xfs_errors: "30"
      swap: ''
relations:
  - ['aws-integrator', 'kubernetes-master']
  - ['aws-integrator', 'kubernetes-worker']
  - [ "filebeat-kube:beats-host", "kubernetes-master:juju-info" ]
  - [ "filebeat-kube:beats-host", "kubernetes-worker:juju-info" ]
  - [ "filebeat:beats-host", "easyrsa:juju-info" ]
  - [ "filebeat:beats-host", "etcd:juju-info" ]
  - [ "kubernetes-master:juju-info", "telegraf:juju-info" ]
  - [ "kubernetes-worker:juju-info", "telegraf:juju-info" ]
  - [ "easyrsa:juju-info", "nrpe-host:general-info" ]
  - [ "etcd:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "kubeapi-load-balancer:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "kubernetes-master:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "kubernetes-worker:nrpe-external-master", "nrpe-host:nrpe-external-master" ]
  - [ "aws-integrator:juju-info", "nrpe-host:general-info" ]

Deploy Charmed Kubernetes

juju add-model cdk
juju deploy charmed-kubernetes --overlay overlay.yaml
juju trust aws-integrator
juju relate filebeat:logstash lma.graylog-beats
juju relate filebeat-kube:logstash lma.graylog-beats
juju relate nrpe-host:monitors lma.nagios-monitors
juju relate telegraf:prometheus-client lma.prometheus-target

Bootstrap Juju on Charmed Kubernetes

Before bootstrapping the CDK Cloud, it’s important to wait until CDK is up and running. When the kubernetes-master message shows “Kubernetes master running” means that it is ready to go . This can take 20 minutes.

watch -c juju status kubernetes-master --color

Model  Controller  Cloud/Region   Version  SLA          Timestamp
cdk    aws-osm-ha  aws/us-east-1  2.5.8    unsupported  10:58:10+02:00

App                Version  Status  Scale  Charm              Store       Rev  OS      Notes
containerd                  active      2  containerd         jujucharms   20  ubuntu 
flannel            0.10.0   active      2  flannel            jujucharms  438  ubuntu 
kubernetes-master  1.15.3   active      2  kubernetes-master  jujucharms  724  ubuntu 

Unit                  Workload  Agent      Machine  Public address  Ports     Message
kubernetes-master/0   active    idle       6        18.233.154.213  6443/tcp  Kubernetes master running.
  containerd/5        active    idle                18.233.154.213            Container runtime available.
  flannel/5           active    idle                18.233.154.213            Flannel subnet 10.1.80.1/24
kubernetes-master/1*  active    idle       7        54.172.90.177   6443/tcp  Kubernetes master running.
  containerd/4        active    idle                54.172.90.177             Container runtime available.
  flannel/4           active    idle                54.172.90.177             Flannel subnet 10.1.20.1/24

Machine  State    DNS             Inst id              Series  AZ          Message
6        started  18.233.154.213  i-038627dbcea0975ce  bionic  us-east-1a  running
7        started  54.172.90.177   i-01702fdcc78a78ee6  bionic  us-east-1b  running

Charmed Kubernetes Credentials

Get kubernetes credentials with the following commands.

mkdir ~/.kube
juju scp kubernetes-master/0:config ~/.kube/config

Configure storage

These additional commands are needed to create storages for CDK to be working in AWS.

# Create a storage class using the `kubernetes.io/aws-ebs` provisioner
kubectl create -f - <<EOY
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-1
provisioner: kubernetes.io/aws-ebs
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp2
EOY

Bootstrap

Make sure to specify the right region. See the previous juju status command to double-check it.

# Bootstrap k8s cloud
REGION=aws/us-east-1
cat ~/.kube/config | juju add-k8s k8s-cloud --local --region=$REGION
juju bootstrap k8s-cloud

# Add a new model for OSM
juju add-model osm

# Create persistent storage volumes
juju create-storage-pool osm-pv kubernetes
juju create-storage-pool packages-pv kubernetes

Install OSM

Choose how you would like Charmed OSM to be deployed.

Standalone

The standalone version is perfect for evaluation and development purposes.

juju deploy osm
juju add-relation lcm-k8s aws-osm-ha:admin/vca.osm-vca

High-Availability

For production use, we offer a high-availability version of Charmed OSM. Each component will be deployed in clusters of three units setup with failover, and requires significantly more resources to operate.

juju deploy osm-ha
juju add-relation lcm-k8s aws-osm-ha:admin/vca.osm-vca

Status

It can take several minutes or longer to install. To monitor the progress of the installation, you can watch the output of juju status:

$ watch -c juju status --color
Every 2.0s: juju status --color                                                                                                                                                                                                                                                         

Model  Controller           Cloud/Region         Version  SLA          Timestamp
osm    k8s-cloud-us-east-1  k8s-cloud/us-east-1  2.6.6    unsupported  12:26:14+02:00

SAAS     Status  Store       URL
osm-vca  active  aws-osm-ha  admin/vca.osm-vca

App             Version  Status  Scale  Charm           Store       Rev  OS          Address         Notes
grafana-k8s              active      3  grafana-k8s     jujucharms   15  kubernetes  10.152.183.94
kafka-k8s                active      3  kafka-k8s       jujucharms    1  kubernetes  10.152.183.31
lcm-k8s                  active      3  lcm-k8s         jujucharms   21  kubernetes  10.152.183.2
mariadb-k8s              active      1  mariadb-k8s     jujucharms   16  kubernetes  10.152.183.246
mon-k8s                  active      3  mon-k8s         jujucharms   14  kubernetes  10.152.183.134
mongodb-k8s              active      3  mongodb-k8s     jujucharms   15  kubernetes  10.152.183.93
nbi-k8s                  active      3  nbi-k8s         jujucharms   24  kubernetes  10.152.183.229
osmclient-k8s            active      1  osmclient-k8s   jujucharms    0  kubernetes  10.152.183.125
pol-k8s                  active      3  pol-k8s         jujucharms   14  kubernetes  10.152.183.130
prometheus-k8s           active      3  prometheus-k8s  jujucharms   14  kubernetes  10.152.183.113
ro-k8s                   active      1  ro-k8s          jujucharms   20  kubernetes  10.152.183.34
ui-k8s                   active      3  ui-k8s          jujucharms   28  kubernetes  10.152.183.232
zookeeper-k8s            active      3  zookeeper-k8s   jujucharms   16  kubernetes  10.152.183.106

Unit               Workload  Agent  Address      Ports                                Message
grafana-k8s/0*     active    idle   10.1.94.8    3000/TCP                             configured
grafana-k8s/1      active    idle   10.1.71.10   3000/TCP                             configured
grafana-k8s/2      active    idle   10.1.33.14   3000/TCP                             configured
kafka-k8s/0*       active    idle   10.1.71.7    9092/TCP                             configured
kafka-k8s/1        active    idle   10.1.94.5    9092/TCP                             configured
kafka-k8s/2        active    idle   10.1.100.32  9092/TCP                             configured
lcm-k8s/0*         active    idle   10.1.71.15   80/TCP                               configured
lcm-k8s/1          active    idle   10.1.94.11   80/TCP                               configured
lcm-k8s/2          active    idle   10.1.100.39  80/TCP                               configured
mariadb-k8s/0*     active    idle   10.1.71.14   3306/TCP,4444/TCP,4567/TCP,4568/TCP  ready
mon-k8s/0*         active    idle   10.1.71.11   8000/TCP                             configured
mon-k8s/1          active    idle   10.1.94.9    8000/TCP                             configured
mon-k8s/2          active    idle   10.1.100.34  8000/TCP                             configured
mongodb-k8s/0*     active    idle   10.1.71.5    27017/TCP                            configured
mongodb-k8s/1      active    idle   10.1.94.3    27017/TCP                            configured
mongodb-k8s/2      active    idle   10.1.100.30  27017/TCP                            configured
nbi-k8s/0*         active    idle   10.1.71.9    9999/TCP                             configured
nbi-k8s/1          active    idle   10.1.94.7    9999/TCP                             configured
nbi-k8s/2          active    idle   10.1.100.35  9999/TCP                             configured
osmclient-k8s/0*   active    idle   10.1.33.15   22/TCP                               ready
pol-k8s/0*         active    idle   10.1.71.8    80/TCP                               configured
pol-k8s/1          active    idle   10.1.94.6    80/TCP                               configured
pol-k8s/2          active    idle   10.1.100.33  80/TCP                               configured
prometheus-k8s/0*  active    idle   10.1.71.12   9090/TCP                             configured
prometheus-k8s/1   active    idle   10.1.94.10   9090/TCP                             configured
prometheus-k8s/2   active    idle   10.1.100.36  9090/TCP                             configured
ro-k8s/0*          waiting   idle   10.1.71.16   9090/TCP                             configured
ui-k8s/0*          active    idle   10.1.33.16   80/TCP                               configured
ui-k8s/1           active    idle   10.1.71.17   80/TCP                               configured
ui-k8s/2           active    idle   10.1.94.12   80/TCP                               configured
zookeeper-k8s/0*   active    idle   10.1.71.6    2181/TCP,2888/TCP,3888/TCP           configured
zookeeper-k8s/1    active    idle   10.1.94.4    2181/TCP,2888/TCP,3888/TCP           configured
zookeeper-k8s/2    active    idle   10.1.100.31  2181/TCP,2888/TCP,3888/TCP           configured

HOW TO

Switch between juju controllers

Charmed Kubernetes:
juju switch aws-osm-ha:admin/cdk

LMA Stack:
juju switch aws-osm-ha:admin/lma

VCA:
juju switch aws-osm-ha:admin/vca

Charmed OSM:
juju switch k8s-cloud-us-east-1:admin/osm

Expose UI

Take the IP of one of the kubernetes-workers (p.e. 3.84.169.68), and execute the following commands:

juju config ui-k8s juju-external-hostname=osm.3.84.169.68.xip.io
juju expose ui-k8s

The ingress module uses nginx. By default, it has the option proxy-body-size to 1m. This will be a problem if a VNF package of more than 1m is uploaded. To solve it, we only have to add an annotation to the ingress.

kubectl -n osm edit ingress ui-k8s

# Add the following line in the annotations
    nginx.ingress.kubernetes.io/proxy-body-size: "0"

You can access now the OSM UI: http://osm.18.215.124.61.xip.io

Expose Prometheus

Take the IP of one of the kubernetes-workers (p.e. 3.84.169.68), and execute the following commands:

juju config prometheus-k8s juju-external-hostname=prometheus.3.84.169.68.xip.io
juju expose prometheus-k8s

You can access now the Prometheus UI: http://prometheus.18.215.124.61.xip.io

Expose Grafana

Take the IP of one of the kubernetes-workers (p.e. 3.84.169.68), and execute the following commands:

juju config grafana-k8s juju-external-hostname=grafana.3.84.169.68.xip.io
juju expose grafana-k8s

You can access now the Grafana UI: http://grafana.18.215.124.61.xip.io

Scale applications

Some applications such as MariaDB and RO (OSM) need to be scaled after the deployment is finished. We are working on improving that, but in the meantime, you should execute the following commands:

  • Scale MariaDB cluster
juju scale-application mariadb-k8s 3
  • Scale RO
juju scale-application ro-k8s 3

Clean up

The easiest way to clean up everything is executing the following commands:

juju kill-controller aws-osm-ha -t 0 -y
juju unregister k8s-cloud-us-east-1

OSM R6 Documentation