[Feature Request]: ElasticSearchIO.Write to allow adding context to input json
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What would you like to happen?
ElasticSearchIO.Write accepts a PCollection. This is the exact JSON that will be written to ElasticSearch. Consequently functions like IndexFn have only (the parsed version of) this JSON as input.
I have a problem where, among others, the exact same JSON get's written to 2 different indexes in different ways. This means I have to create multiple instances of ElasticSearchIO.Write, one for each permutation of settings. It's not a major problem because there are only 7 permutations in my case but it could be many more. It does lead to a big process graph in dataflow and also it leads many small batches because there is a latency constraint,
The reason is that there is no way of adding some context or meta data to that JSON string. I could add an extra field to that json but I have no way of removing it before it gets written to the index.
A possible solution would be to have a type parameter for the input and a settable jsonFn that extracts the json string to be written.
### Issue Priority
Priority: 3
### Issue Component
Component: io-java-elasticsearch
Contributor guide
Assessment
This issue has not been assessed yet.