Published on

Using Elasticdump to Backup Elasticsearch Indexes to JSON

Authors

We will use Elasticdump to dump data from Elasticsearch to json files on disk, then delete the index, then restore data back to elasticsearch

ruanbekker-cheatsheets

Install Elasticdump:

$ docker run -it node:alpine sh
$ npm install elasticdump -g

Create a Index:

$ curl -XPUT http://10.79.2.193:9200/test-index
{"acknowledged":true}

Ingest Some Data into the Index:

$ curl -XPUT http://10.79.2.193:9200/test-index/docs/doc1 -d '{"name": "ruan", "age": 30}'
{"_index":"test-index","_type":"docs","_id":"doc1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

$ curl -XPUT http://10.79.2.193:9200/test-index/docs/doc2 -d '{"name": "stefan", "age": 29}'
{"_index":"test-index","_type":"docs","_id":"doc2","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}

Elasticdump to dump the ata

First dump the mappings:

$ elasticdump --input=http://10.79.2.193:9200/test-index --output=/opt/backup/elasticsearch/es_test-index_mapping.json --type=mapping
Mon, 26 Jun 2017 14:15:34 GMT | starting dump
Mon, 26 Jun 2017 14:15:34 GMT | got 1 objects from source elasticsearch (offset: 0)
Mon, 26 Jun 2017 14:15:34 GMT | sent 1 objects to destination file, wrote 1
Mon, 26 Jun 2017 14:15:34 GMT | got 0 objects from source elasticsearch (offset: 1)
Mon, 26 Jun 2017 14:15:34 GMT | Total Writes: 1
Mon, 26 Jun 2017 14:15:34 GMT | dump complete

Then dump the data:

$ elasticdump --input=http://10.79.2.193:9200/test-index --output=/opt/backup/elasticsearch/es_test-index.json --type=data
Mon, 26 Jun 2017 14:15:43 GMT | starting dump
Mon, 26 Jun 2017 14:15:43 GMT | got 2 objects from source elasticsearch (offset: 0)
Mon, 26 Jun 2017 14:15:43 GMT | sent 2 objects to destination file, wrote 2
Mon, 26 Jun 2017 14:15:43 GMT | got 0 objects from source elasticsearch (offset: 2)
Mon, 26 Jun 2017 14:15:43 GMT | Total Writes: 2
Mon, 26 Jun 2017 14:15:43 GMT | dump complete

Preview the Metadata

$ cat /opt/backup/elasticsearch/es_test-index_mapping.json | python -m json.tool
{
    "test-index": {
        "mappings": {
            "docs": {
                "properties": {
                    "age": {
                        "type": "long"
                    },
                    "name": {
                        "type": "string"
                    }
                }
            }
        }
    }
}

Preview the Data

$ cat /opt/backup/elasticsearch/es_test-index.json | jq
{
  "_index": "test-index",
  "_type": "docs",
  "_id": "doc1",
  "_score": 1,
  "_source": {
    "name": "ruan",
    "age": 30
  }
}
{
  "_index": "test-index",
  "_type": "docs",
  "_id": "doc2",
  "_score": 1,
  "_source": {
    "name": "stefan",
    "age": 29
  }
}

Restore The Data

Let's test the restoring part, go ahead and delete The index:

$ curl -XDELETE http://10.79.2.193:9200/test-index
{"acknowledged":true}

Restore the Index by Importing the Mapping:

$ elasticdump --input=/opt/backup/elasticsearch/es_test-index_mapping.json --output=http://10.79.2.193:9200/test-index --type=mapping
Mon, 26 Jun 2017 14:51:48 GMT | starting dump
Mon, 26 Jun 2017 14:51:48 GMT | got 1 objects from source file (offset: 0)
Mon, 26 Jun 2017 14:51:48 GMT | sent 1 objects to destination elasticsearch, wrote 1
Mon, 26 Jun 2017 14:51:48 GMT | got 0 objects from source file (offset: 1)
Mon, 26 Jun 2017 14:51:48 GMT | Total Writes: 1
Mon, 26 Jun 2017 14:51:48 GMT | dump complete

Verify to see if the Index Exist:

$ curl -s -XGET http://10.79.2.193:9200/_cat/indices?v | grep -E "(docs.count|test)"
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test-index                  5   1          0            0       650b           650b

Restore the Data for the Index:

Use elasticdump to restore the data from json to elasticsearch:

$ elasticdump --input=/opt/backup/elasticsearch/es_test-index.json --output=http://10.79.2.193:9200/test-index --type=data
Mon, 26 Jun 2017 14:53:56 GMT | starting dump
Mon, 26 Jun 2017 14:53:56 GMT | got 2 objects from source file (offset: 0)
Mon, 26 Jun 2017 14:53:56 GMT | sent 2 objects to destination elasticsearch, wrote 2
Mon, 26 Jun 2017 14:53:56 GMT | got 0 objects from source file (offset: 2)
Mon, 26 Jun 2017 14:53:56 GMT | Total Writes: 2
Mon, 26 Jun 2017 14:53:56 GMT | dump complete

Verify to see if the Documents was Ingested:

$ curl -s -XGET http://10.79.2.193:9200/_cat/indices?v | grep -E '(docs.count|test)'
health status index                     pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test-index                  5   1          2            0       650b           650b

Preview the Data from Elasticsearch:

$ curl -s -XGET http://10.79.2.193:9200/test-index/_search?pretty

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "test-index",
      "_type" : "docs",
      "_id" : "doc1",
      "_score" : 1.0,
      "_source" : {
        "name" : "ruan",
        "age" : 30
      }
    }, {
      "_index" : "test-index",
      "_type" : "docs",
      "_id" : "doc2",
      "_score" : 1.0,
      "_source" : {
        "name" : "stefan",
        "age" : 29
      }
    } ]
  }
}

Resources:

Thank You

Thanks for reading, feel free to check out my website, feel free to subscribe to my newsletter or follow me at @ruanbekker on Twitter.

Discuss on Twitter • View on GitHub