apache / apache/beam

Allow modification of JSON value before writing to ElasticSearch

Open
#19,182 0 comments 0 reactions 0 assignees View on GitHub
elasticsearch improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I have an Apache Beam streaming job which reads data from Kafka and writes to ElasticSearch using ElasticSearchIO.

The issue I'm having is that messages in Kafka already have _`key`_ field, and using `ElasticSearchIO.Write.withIdFn()` I'm mapping this field to document _`_id`_ field in ElasticSearch.

Having a big volume of data I don't want the _`key`_ field to be also written to ElasticSearch as part of _`_source`_.

Is there an option/workaround that would allow doing that?

Imported from Jira [BEAM-6168](https://issues.apache.org/jira/browse/BEAM-6168). Original Jira may contain additional context.
Reported by: marknorkin.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.