llnl / llnl/elastic-stacker

Load and dump index contents

Open
#16 0 comments 0 reactions 1 assignee Claimed by @taliaferro View on GitHub
Dominant language
Python
Stars
7
Forks
1
PR merge metrics
No merged PRs in 30d

Description

In some cases (particularly when working across an airgap) it is useful to dump and import the entire contents of an Elasticsearch index wholesale.
This should not be included in the system_dump (dumping every single document in the cluster is obviously a very time-consuming process) but rather a separate operation to be run on smaller indices (for example, indices used as part of enrich processors.)

For inspiration, look to TaskRabbit's [elasticsearch-dump](https://github.com/elasticsearch-dump/elasticsearch-dump) which does exactly this. It uses the [Scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html) which Elasticsearch includes precisely to enable these kinds of large data processing workloads.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.