Allow for Parent parameter for Parent/Child join - ElasticsearchIO
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
It is impossible to index Parent/Child documents using ElasticsearchIO
When using parent/child documents in Elasticsearch, child documents must be indexed using the "parent" attribute when using the bulk endpoint.
[https://www.elastic.co/guide/en/elasticsearch/guide/master/indexing-parent-child.html](https://www.elastic.co/guide/en/elasticsearch/guide/master/indexing-parent-child.html)
[https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)
[https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java](https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java)
There needs to be a "parent" attribute on the DocumentMetadata object. I think a FieldValueExtractFn would work really nicely for this
Imported from Jira [BEAM-11283](https://issues.apache.org/jira/browse/BEAM-11283). Original Jira may contain additional context.
Reported by: acuitymd.
Contributor guide
Research direction
Start with sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java and inspect DocumentMetadata, FieldValueExtractFn, and the bulk indexing path. The change is complete when child documents can supply a parent value and Elasticsearch bulk requests include the parent attribute; use the existing ElasticsearchIO tests if available to verify this behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- data, search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100