apache / apache/beam

[Feature Request]: ElasticSearchIO.Write to allow adding context to input json

Open
#22,840 6 comments 0 reactions 0 assignees View on GitHub
elasticsearch io java new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

ElasticSearchIO.Write accepts a PCollection. This is the exact JSON that will be written to ElasticSearch. Consequently functions like IndexFn have only (the parsed version of) this JSON as input.

I have a problem where, among others, the exact same JSON get's written to 2 different indexes in different ways. This means I have to create multiple instances of ElasticSearchIO.Write, one for each permutation of settings. It's not a major problem because there are only 7 permutations in my case but it could be many more. It does lead to a big process graph in dataflow and also it leads many small batches because there is a latency constraint,

The reason is that there is no way of adding some context or meta data to that JSON string. I could add an extra field to that json but I have no way of removing it before it gets written to the index.

A possible solution would be to have a type parameter for the input and a settable jsonFn that extracts the json string to be written.

### Issue Priority

Priority: 3

### Issue Component

Component: io-java-elasticsearch

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.