Elasticsearch Kubernetes Charm


#1

I’m trying to build an Elastic search Kubernetes Series Charm. Elasticsearch needs a bigger file descriptor limit than the default one, setting this limit in a docker image is supported but unfortunately not in Kubernetes as this Issue describes: https://github.com/kubernetes/kubernetes/issues/3595

Nonetheless there are some workarounds for this:
https://github.com/kubernetes/kubernetes/issues/3595#issuecomment-288451522

So after creating my own docker image and adapting my spec_template I get the follwing error when I try to deploy the charm:
ubuntu@osm2:~/canonical-osm/charms/layers$ kubectl -n osm logs elasticsearch-k8s-0
bin/custom-entrypoint.sh: line 5: ulimit: open files: cannot modify limit: Operation not permitted

Parting from my limited kubernetes knowledge it seems to me that the securityContext options aren’t taking effect. Is what I’m trying to achieve even possible with the current state of kubernetes charms?

Dockerfile:

FROM docker.elastic.co/elasticsearch/elasticsearch-oss:6.4.2
      
WORKDIR /usr/share/elasticsearch

USER root

COPY configs/elasticsearch.yml config/
COPY custom-entrypoint.sh bin/

VOLUME /usr/share/elasticsearch/data

RUN chown elasticsearch:elasticsearch config/elasticsearch.yml config/log4j2.properties bin/custom-entrypoint.sh && \
     chmod 0750 bin/custom-entrypoint.sh

CMD ["/bin/bash","bin/custom-entrypoint.sh"]

custom-entrypoint.sh:

 #!/bin/bash
      
 set -e

 ulimit -n 65536

 ulimit -u 2048

 #Set memlock limit
 ulimit -l unlimited

 #Call original entrypoint script
 #Maybe su elasticsearch bin/es-docker "$@"
 exec /docker-entrypoint.sh "${@}"

spec_template.yaml:

    containers:
    - name: %(name)s
      imageDetails:
        imagePath: %(docker_image_path)s
        username: %(docker_image_username)s
        password: %(docker_image_password)s
      ports:
      - containerPort: %(advertised-port)s
        protocol: TCP
      environment:
        ES_JAVA_OPTS: "-Xmx256m -Xms256m"
      networks:
      - elk
      config:
        ALLOW_ANONYMOUS_LOGIN: "yes"
      securityContext:
        privileged: true
        capabilities:
          add: 
          - IPC_LOCK
          - SYS_RESOURCE

#2

What version of Juju are you using?

Recent 2.5 releases support security context at a container level (which I think is what you want).
2.6 beta 1 supports security context for the pod.

What you have looks syntactically valid. Can you use kubectl to inspect the deployed pod just to make sure that the container spec is being correctly set up by Juju to have the specified security context?


#3

I’m using Juju version 2.5.4.
I attached the output of the command kubectl describe pod there doesn’t seem to be anything about security context.

ubuntu@osm2:~/canonical-osm$ kubectl -n osm describe pod elasticsearch-k8s-0
Name:               elasticsearch-k8s-0
Namespace:          osm
Priority:           0
PriorityClassName:  <none>
Node:               osm2/10.55.60.82
Start Time:         Wed, 03 Apr 2019 12:47:21 +0000
Labels:             controller-revision-hash=elasticsearch-k8s-8568d5889b
                    juju-application=elasticsearch-k8s
                    juju-controller-uuid=6189229e-16df-4aec-820d-4d98de3e1ae4
                    juju-model-uuid=378c9365-26d7-4a65-839a-cf25066c666c
                    statefulset.kubernetes.io/pod-name=elasticsearch-k8s-0
Annotations:        <none>
Status:             Running
IP:                 10.1.1.104
Controlled By:      StatefulSet/elasticsearch-k8s
Containers:
  elasticsearch-k8s:
    Container ID:   docker://4a0cbcb2106222fd89528ed41773da66d82a12d5e96f5d6b5b1a41de33e115ca
    Image:          domfleischmann/elasticsearch:6.4.2
    Image ID:       docker-pullable://domfleischmann/elasticsearch@sha256:82868b3f10857c90884191cc17ffadc0345520a7d7eaa7c7566cb5f33942608a
    Port:           9200/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 04 Apr 2019 08:05:52 +0000
      Finished:     Thu, 04 Apr 2019 08:05:53 +0000
    Ready:          False
    Restart Count:  233
    Environment:
      ALLOW_ANONYMOUS_LOGIN:  yes
    Mounts:
      /usr/share/elasticsearch/config/yml from elasticsearch-k8s-es-yaml-config (rw)
      /usr/share/elasticsearch/data from database-f6d70f6e (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-bvknq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  database-f6d70f6e:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  database-f6d70f6e-elasticsearch-k8s-0
    ReadOnly:   false
  elasticsearch-k8s-es-yaml-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      elasticsearch-k8s-es-yaml-config
    Optional:  false
  default-token-bvknq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-bvknq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                     From           Message
  ----     ------   ----                    ----           -------
  Warning  BackOff  8m55s (x4207 over 15h)  kubelet, osm2  Back-off restarting failed container
  Normal   Pulled   3m59s (x181 over 15h)   kubelet, osm2  Container image "domfleischmann/elasticsearch:6.4.2" already present on machine

#4

After trying juju 2.6 it seems to be working!


#5

Sorry yes, I got the versions mixed up. I just finished running up a couple of systems to test.

juju 2.5.4 supports pod level security contexts.
In juju 2.6, we added support for container level security contexts as well.


#6

Great to hear! :medal_sports:

Please feel free to ask any further questions if you get stuck again :slight_smile: