elastic / elastic/logstash

Add ingest migrator CLI tool

Open
#7,329 2 comments 0 reactions 0 assignees View on GitHub
ingest-migration
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Based on all the great work that @original-brownbear has done in `tools/ingest-convertor` it is time now to create a CLI tool that provides an end-user interface to move from ingest pipeline to LS config.

I propose two flavors to make it dead simple and easy for our users -- offline and an online.

### Offline

```sh
bin/ingest-convert.sh --input file:///tmp/apache.json --output file:///tmp/pipeline.conf
```
This would call the `ingest_pipeline_to_logstash()` js function as done in https://github.com/elastic/logstash/blob/master/tools/ingest-converter/src/main/java/org/logstash/ingest/Pipeline.java

We can create all the heavy lifting in Java and just write a shell wrapper around it.

### Online (Lower priority)

This assumes the user has access to the ES endpoint where the ingest pipeline is defined.

```sh
bin/ingest-convert.sh --input apache --output /tmp/pipeline.conf
```
defaults to `localhost:9200`. It then gets the json pipeline using the ES API call `GET _ingest/pipeline/apache`, stores it in memory or tmp location and then converts it using the utilities created.

can accept optional ES config for credentials:

```
bin/ingest-convert.sh --pipeline apache --output /tmp/pipeline.conf --user foo --password
```

For this, maybe we can use the Java high-level client from ES (when it gets shipped)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.