Writing a Kubernetes charm


#1

Introduction

Kubernetes charms are the similar as traditional cloud charms. The same model is used. An application has units and each unit relation has its own data bag. The same hooks are invoked but unlike traditional charms there’s no expectation that the install hook be implemented. This is less relevant anyway using the reactive framework.

The only mandatory task for the charm is to tell Juju some key pod / container configuration. Kubernetes charms are recommended to be reactive and there’s a base layer that is similar to the base layer used for traditional charms:

https://github.com/juju-solutions/layer-caas-base

The basic flow for how the charm operates is:

  1. charm calls config-get to retrieve the charm settings from Juju

  2. charm translates settings to create pod configuration

  3. charm calls pod-spec-set to tell Juju how to create pods/units

  4. charm can use status-set or juju-log or any other hook command the same as for traditional charms

  5. charm can implement hooks the same as for traditional charms

There’s no need for the charm to apt install anything - the operator docker image has all the necessary reactive and charm helper libraries baked in.

The charm can call pod-spec-set at any time and Juju will update any running pods with the new pod spec. This may be done in response to the config-changed hook due to the user changing charm settings, or when relations are joined etc. Juju will check for actual changes before restarting pods so the call is idempotent.

Note: the pod spec applies for the application as a whole. All pods are homogeneous.

Sample gitlab, mariadb charms are at https://github.com/wallyworld/caas. These charms are POC only and are not production quality.

There’s also Kubeflow charms in development at https://github.com/juju-solutions.

Kubernetes charm store

All Kubernetes charms ready for deployment are hosted at the staging charm store. This is all work in progress. You can find some early prototype pre-built Kubernetes series charms and bundles here:

Container images

Charms specify what container image to use by including a resource definition.

resources:
  mysql_image:
    type: oci-image
    description: Image used for mysql pod.

oci-image is a new type of charm resource (we already have file).

The image is pushed along with the charm and hosted by the staging charm store. Standard Juju resource semantics apply. A charm is released (published) as a tuple of (charm revision, resource version). This allows the charm and associated image to be published as a known working configuration.

Example workflow

To build and push a charm to the staging charm store, ensure you have the edge charm snap installed.
After hacking on the charm and running charm build to fully generate it, you push, attach, release:

export JUJU_CHARMSTORE=https://api.staging.jujucharms.com/charmstore
cd <build dir>
charm push . mariadb-k8s
docker pull mariadb
charm attach cs:~wallyworld/mariadb-k8s-8 mysql_image=mariadb
charm release cs:~wallyworld/mariadb-k8s-9 --resource mysql_image-0

See
charm help push
charm help attach

Charms in more detail

Use the importation below in addition to looking at the charms already written to see how this all hangs together.

To illustrate how a charm tells Juju how to configure a unit’s pod, here’s the template YAML snippet used by the Kubernetes mariadb charm. Note the placeholders which are filled in from the charm config obtained via config-get.

ports:
- containerPort: %(port)s
  protocol: TCP
config:
 MYSQL_ROOT_PASSWORD: %(rootpassword)
 MYSQL_USER: %(user)s
 MYSQL_PASSWORD: %(password)s
 MYSQL_DATABASE: %(database)s
files:
 - name: configurations
   mountPath: /etc/mysql/conf.d
   files:
     custom_mysql.cnf: |
       [mysqld]
       skip-host-cache
       skip-name-resolve         
       query_cache_limit = 1M
       query_cache_size = %(query-cache-size)s

The charm simply sends this YAML snippet to Juju using the pod_spec_set() charm helper.
Here’s a code snippet from the mariadb charm.

from charms.reactive import when, when_not
from charms.reactive.flags import set_flag, get_state, clear_flag
from charmhelpers.core.hookenv import (
    log,
    metadata,
    status_set,
    config,
    network_get,
    relation_id,
)

from charms import layer

@when_not('layer.docker-resource.mysql_image.fetched')
def fetch_image():
    layer.docker_resource.fetch('mysql_image')

@when('mysql.configured')
def mariadb_active():
    status_set('active', '')

@when('layer.docker-resource.mysql_image.available')
@when_not('mysql.configured')
def config_mariadb():
    status_set('maintenance', 'Configuring mysql container')

    spec = make_pod_spec()
    log('set pod spec:\n{}'.format(spec))
    layer.caas_base.pod_spec_set(spec)

    set_flag('mysql.configured')
....

Important Difference With Cloud Charms

Charms such as databases which have a provides endpoint often need to set in relation data the IP address to which related charms can connect. The IP address is obtained using network-get, often something like this:

@when('mysql.configured')
@when('server.database.requested')
def provide_database(mysql):
    info = network_get('server', relation_id())

    for request, application in mysql.database_requests().items():
        database_name = get_state('database')
        user = get_state('user')
        password = get_state('password')

        mysql.provide_database(
            request_id=request,
            host=host,
            port=3306,
            database_name=database_name,
            user=user,
            password=password,
        )
        clear_flag('server.database.requested')

Workload Status

Currently, there’s no well defined way for a Kubernetes charm to query the status of the workload it is managing. So although the charm can reasonably set status as say blocked when it’s waiting for a required relation to be created, or maintenance when the pod spec is being set up, there’s no real way for the charm to know when to set active.

Juju helps solve this problem by looking at the pod status and uses that in conjunction with the status reported by the charm to determine what to display to the user. Workload status values of waiting, blocked, maintenance, or any error conditions, are always reported directly. However, if the charm sets status as active, this is not shown as such until the pod is reported as Running. So all the charm has to do is set status as active when all of its initial setup is complete and the pod spec has been sent to Juju, and Juju will “Do The Right Thing” from then on. Both the gitlab and mariadb sample charms illustrate how workload status can set correctly set.

A future enhancement will be to allow the charm to directly query the workload status and the above workaround will become unnecessary.

Kubernetes Specific Pod Config

It’s possible to specify Kubernetes specific pod configuration in the pod spec YAML created by the charm. The supported attributes are:

  • livenessProbe
  • readinessProbe
  • imagePullPolicy

The syntax used is standard k8s pod spec syntax.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

Here’s an example:

containers:
  - name: gitlab
    imagePullPolicy: Always
    ports:
    - containerPort: 80
      protocol: TCP
    livenessProbe:
      initialDelaySeconds: 10
      httpGet:
        path: /ping
        port: 8080
    readinessProbe:
      initialDelaySeconds: 10
      httpGet:
        path: /pingReady
        port: www
    config:
      attr: foo=bar; fred=blogs

Spark Charm -> Spark 2.4
#2

Juju 2.5.0 Beta 3 Release Notes