ElasticsearchIO listener timeout
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Still getting a listener timeout when trying to insert a large amount of documents.
```
java.io.IOException: listener timeout after waiting for [30000] ms
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:660)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:219)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1226)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1200)
```
The listener timeouts are not handled in the underlying ES RestClient.
A possible solution would be to catch these exceptions and include them in the retry handling mechanism
Imported from Jira [BEAM-5988](https://issues.apache.org/jira/browse/BEAM-5988). Original Jira may contain additional context.
Reported by: wouts.
Contributor guide
Research direction
Start in ElasticsearchIO$Write$WriteFn.flushBatch and finishBundle, where the reported RestClient listener timeout occurs. Review how the existing retry handling treats exceptions, then verify that listener timeouts are included without losing the original failure context and that large document inserts no longer fail immediately after the 30000 ms timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100