[Tutorial] Elasticsearch Backup/Restore Using S3


#1

As a follow up to the previous elasticsearch post I felt it would be helpful to highlight a few useful elasticsearch lifecycle operations around backup/restore using S3.

Configure Elasticsearch to use the S3 repository backend

Set the elasticsearch-keystore values so we can backup/restore a snapshot using an s3 backend.

juju run --application elasticsearch \
    "export ES_PATH_CONF=/etc/elasticsearch && echo -n '<aws-access-key>' | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.access_key"

juju run --application elasticsearch \
    "export ES_PATH_CONF=/etc/elasticsearch && echo -n '<aws-secret-key>' | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.secret_key"

# Restart elasticsearch
juju run --application elasticsearch "service elasticsearch restart"

If using a heterogeneous node deploy this would need to be preformed on all master and data node types. E.g.

# Master nodes
juju run --application es-master \
    "export ES_PATH_CONF=/etc/elasticsearch && echo -n '<aws-access-key>' | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.access_key"

juju run --application es-master \
    "export ES_PATH_CONF=/etc/elasticsearch && echo -n '<aws-secret-key>' | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.secret_key"

# Data nodes
juju run --application es-data \
    "export ES_PATH_CONF=/etc/elasticsearch && echo -n '<aws-access-key>' | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.access_key"

juju run --application es-data \
    "export ES_PATH_CONF=/etc/elasticsearch && echo -n '<aws-secret-key>' | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.secret_key"


# Restart elasticsearch on both node types
juju run --application es-master "service elasticsearch restart"
juju run --application es-data "service elasticsearch restart"

Setup the snapshot repository

curl -X PUT "http://<elasticsearch-node-ip>:9200/_snapshot/<repository-name>" -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "<s3-bucket-name>",
    "region": "<region-if-applicable>"
  }
}'

Backup a snapshot of one or more indices to an S3 repository

curl -X PUT "http://<elasticsearch-node-ip>:9200/_snapshot/<repository-name>/<snapshot-name>" -H 'Content-Type: application/json' -d'
{
  "indices": "<indices-to-include-in-snapshot-separated-by-commas>",
  "ignore_unavailable": true,
  "include_global_state": false
}
'

Restore a snapshot of one or more indices from S3 repository

curl -X POST "http://<elasticsearch-node-ip>:9200/_snapshot/<repository-name>/<snapshot-name>/_restore" -H 'Content-Type: application/json' -d'
{
  "indices": "<indices-to-include-in-snapshot-restore-separated-by-commas>",
  "index_settings": {
    "index.number_of_replicas": 0
  },
  "ignore_index_settings": [
    "index.refresh_interval"
  ]
}
'

Modify index settings

Example of how to modify index settings.

curl -X PUT "http://<elasticsearch-node-ip>:9200/<index-name>/_settings" -H 'Content-Type: application/json' -d'
{
    "index" : {
        "number_of_replicas" : 1
    }
}'

Juju 2.6.1 Release Notes
WIP - Juju 2.6.1 Release Notes
[Tutorial] Heterogeneous Elasticsearch Deploy Using Juju