[Tutorial] Heterogeneous Elasticsearch Deploy Using Juju


#1

Running through another Elasticsearch deploy this weekend, I thought I would jot it down so others can see/use it too.

This walkthrough uses Juju to deploy a heterogeneous Elasticsearch cluster consisting of:

* 18 Data Nodes
* 3 Master Nodes
* 1 Ingest Node

Note: My ingest use case is minimal so I only deploy a single ingest node, but could have deployed many more if needed.

Deploy and Relate the Elasticsearch Components

# Master Nodes
juju deploy cs:~omnivector/elasticsearch es-master -n3 \
    --constraints "instance-type=t3.2xlarge root-disk=20G" \
    --config java-opts="-Xms16g -Xmx16g" --config cluster-name="es-bdx-testing" \
    --config node-type=master

# Data Nodes
juju deploy cs:~omnivector/elasticsearch es-data -n18 \
    --constraints "instance-type=t3.2xlarge root-disk=20G" \
    --config java-opts="-Xms16g -Xmx16g" --config cluster-name="es-bdx-testing" \
    --storage data=ebs-ssd,500G --config node-type=data

# Ingest Node(s)
 juju deploy cs:~omnivector/elasticsearch es-ingest -n1 \
    --constraints "instance-type=t3.2xlarge root-disk=20G" \
    --config java-opts="-Xms16g -Xmx16g" --config cluster-name="es-bdx-testing" \
    --config node-type=ingest

# Relate 

juju relate es-master:provide-master es-data:require-master
juju relate es-master:provide-master es-ingest:require-master

At this point you should have an Elasticsearch cluster running with 3 master, 18 data, and 1 ingest node.

Deploy and Relate Kibana

juju deploy cs:~omnivector/kibana \
    --constraints "instance-type=t3.medium root-disk=20G"

juju relate kibana:elasticsearch es-master:client

Final Result

Model             Controller  Cloud/Region   Version  SLA          Timestamp
my-bdx-es-dev-00  pdl-aws     aws/us-west-2  2.5.4    unsupported  14:44:17Z

App        Version  Status  Scale  Charm          Store       Rev  OS      Notes
es-data    6.7.2    active     18  elasticsearch  jujucharms   37  ubuntu  exposed
es-ingest  6.7.2    active      1  elasticsearch  jujucharms   37  ubuntu  
es-master  6.7.2    active      3  elasticsearch  jujucharms   37  ubuntu  exposed
kibana     6.7.2    active      1  kibana         jujucharms    7  ubuntu  exposed

Unit          Workload  Agent  Machine  Public address  Ports              Message
es-data/27    active    idle   37       172.31.104.38   9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/28    active    idle   38       172.31.102.221  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/29    active    idle   39       172.31.102.184  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/30    active    idle   40       172.31.104.57   9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/31    active    idle   41       172.31.102.130  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/32    active    idle   42       172.31.102.220  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/33    active    idle   43       172.31.102.107  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/34*   active    idle   44       172.31.104.104  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/35    active    idle   45       172.31.104.162  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/36    active    idle   50       172.31.104.13   9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/37    active    idle   51       172.31.102.153  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/38    active    idle   52       172.31.104.8    9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/39    active    idle   53       172.31.103.59   9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/40    active    idle   54       172.31.104.166  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/41    active    idle   55       172.31.102.30   9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/42    active    idle   56       172.31.102.132  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/43    active    idle   57       172.31.103.155  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-data/44    active    idle   58       172.31.103.112  9200/tcp,9300/tcp  Elasticsearch Running - 18 x data nodes
es-ingest/0*  active    idle   47       172.31.103.119  9200/tcp,9300/tcp  Elasticsearch Running - 1 x ingest nodes
es-master/6*  active    idle   25       172.31.102.234  9200/tcp,9300/tcp  Elasticsearch Running - 3 x master nodes
es-master/7   active    idle   26       172.31.102.93   9200/tcp,9300/tcp  Elasticsearch Running - 3 x master nodes
es-master/8   active    idle   27       172.31.103.64   9200/tcp,9300/tcp  Elasticsearch Running - 3 x master nodes
kibana/3      active    idle   49       172.31.103.57   80/tcp             Kibana available

Machine  State    DNS             Inst id              Series  AZ          Message
25       started  172.31.102.234  i-0f2d5edc6002af708  bionic  us-west-2a  running
26       started  172.31.102.93   i-0cb4d854d21dec9f3  bionic  us-west-2a  running
27       started  172.31.103.64   i-0b5678c06c6f4cab7  bionic  us-west-2b  running
37       started  172.31.104.38   i-0b772836dc61da756  bionic  us-west-2c  running
38       started  172.31.102.221  i-0fa0be0eef3b64d00  bionic  us-west-2a  running
39       started  172.31.102.184  i-01b8d99af0635b58a  bionic  us-west-2a  running
40       started  172.31.104.57   i-0ffd933a2af314de2  bionic  us-west-2c  running
41       started  172.31.102.130  i-09ece2c98bc7ea034  bionic  us-west-2a  running
42       started  172.31.102.220  i-0423e0ea5281eebca  bionic  us-west-2a  running
43       started  172.31.102.107  i-09c6acefdc0d1d89b  bionic  us-west-2a  running
44       started  172.31.104.104  i-09753d46515d30947  bionic  us-west-2c  running
45       started  172.31.104.162  i-0fcdfce2d39951edd  bionic  us-west-2c  running
47       started  172.31.103.119  i-05f46dfd3a29933d7  bionic  us-west-2b  running
49       started  172.31.103.57   i-02c5f015a73ad5bfe  bionic  us-west-2b  running
50       started  172.31.104.13   i-0f839ddbc728ea04c  bionic  us-west-2c  running
51       started  172.31.102.153  i-020b38070c87de9b8  bionic  us-west-2a  running
52       started  172.31.104.8    i-065f61eae901ef600  bionic  us-west-2c  running
53       started  172.31.103.59   i-0ea9dfe9fd0075c66  bionic  us-west-2b  running
54       started  172.31.104.166  i-08ca7dfb1a1460c53  bionic  us-west-2c  running
55       started  172.31.102.30   i-0b6f2eeecf4806135  bionic  us-west-2a  running
56       started  172.31.102.132  i-0670349e0cd3695ae  bionic  us-west-2a  running
57       started  172.31.103.155  i-0e025ba96be105655  bionic  us-west-2b  running
58       started  172.31.103.112  i-07e1412f4619bf527  bionic  us-west-2b  running

As always, pull requests welcome.

Enjoy!


Juju 2.6.1 Release Notes
WIP - Juju 2.6.1 Release Notes
#2

The link to your code at github for layer-elasticsearch points to 404


#3

Wow, that’s a lot of content being indexed.


#4

:+1:

I wrote a quick bit on how I am loading it up here.