apache / apache/beam

ElasticsearchIO listener timeout

Open
#19,211 0 comments 0 reactions 0 assignees View on GitHub
bug elasticsearch io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Still getting a listener timeout when trying to insert a large amount of documents.

```

java.io.IOException: listener timeout after waiting for [30000] ms
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:660)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:219)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1226)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1200)
```

The listener timeouts are not handled in the underlying ES RestClient.

A possible solution would be to catch these exceptions and include them in the retry handling mechanism

Imported from Jira [BEAM-5988](https://issues.apache.org/jira/browse/BEAM-5988). Original Jira may contain additional context.
Reported by: wouts.

Contributor guide

Open the contributing guide

Research direction

Start in ElasticsearchIO$Write$WriteFn.flushBatch and finishBundle, where the reported RestClient listener timeout occurs. Review how the existing retry handling treats exceptions, then verify that listener timeouts are included without losing the original failure context and that large document inserts no longer fail immediately after the 30000 ms timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.