Handle item-level failures when doing bulk ingestion
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 133
- Forks
- 205
- Avg merge
- 16h 3m
- Merged PRs (30d)
- 102
Description
### Problem Description
Connectors contain logic [here](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/es/sink.py#L203) that collects documents to bulk operations and then sends batches into Elasticsearch.
There is a todo in the code:
```
# TODO: retry 429s for individual items here
```
It is possible that the whole request will return 200, but individual items within this request will be marked as failed - e.g. you've sent 500 items and 3 of them returned 429 because EIS or ML node was unable to process them.
So we need to add logic that collects all items that failed ingestion in the batch and retry them a couple times. Actual implementation of this needs to be discusssed.
### Proposed Solution
When bulk request returns successful status but individual items are not successful these items need to be retried - either combined with more items from the queue or just these items alone. If after a couple attempts ingestion fails these items should be dropped with a log line.
### Additional Context
Be mindful of Error Monitor in [_process_bulk_response](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/es/sink.py#L238) - it counts failed items and makes a decision to drop some items, this logic should only execute after retrying of the ingestion happened
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in app/connectors_service/connectors/es/sink.py at the bulk collection logic around line 203 and follow _process_bulk_response around line 238. Determine how individual 429 failures are identified and how retries interact with the Error Monitor. Done means failed items are retried a couple of times before the drop-and-log behavior runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100