apache / apache/beam

Expand aliases in ElasticsearchIO.Read

Open
#19,441 0 comments 0 reactions 0 assignees View on GitHub
elasticsearch improvement io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I am building a pipeline that needs to process all records in a set of indexes, each suffixed with a timestamp. I have an alias that matches all of these indexes at once. However, I cannot use the alias name in ElasticsearchIO as it will try to read stats from this specific index. Because it is an alias and not an actual index, the response contains no count for the alias name itself and therefore Beam (Dataflow?) will estimate the size as being 0. This makes the pipeline end without even executing the query on the alias, even though that would have returned loads of documents.

This should be easy to fix as the results of //_stats only contains indexes references by that alias, so instead of looking for a key in the `indices` key in the returned JSON, it should consider all returned indexes and add the estimated sizes together.

Imported from Jira [BEAM-6920](https://issues.apache.org/jira/browse/BEAM-6920). Original Jira may contain additional context.
Reported by: MadEgg.

Contributor guide

Open the contributing guide

Research direction

Start at ElasticsearchIO.Read and trace how the //_stats response is used to estimate input size. Verify the behavior with an alias covering timestamp-suffixed indexes, then ensure the estimate includes the returned indexes and the pipeline still executes the alias query instead of treating its size as zero.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.