elastic / elastic/connectors

Handle item-level failures when doing bulk ingestion

Open
#4,002 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

effort:low enhancement good first issue priority:medium sdh-driven team:extract-and-transform
Dominant language
Python
Stars
133
Forks
205
Avg merge
16h 3m
Merged PRs (30d)
102

Description

### Problem Description

Connectors contain logic [here](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/es/sink.py#L203) that collects documents to bulk operations and then sends batches into Elasticsearch.

There is a todo in the code:
```
# TODO: retry 429s for individual items here
```

It is possible that the whole request will return 200, but individual items within this request will be marked as failed - e.g. you've sent 500 items and 3 of them returned 429 because EIS or ML node was unable to process them.

So we need to add logic that collects all items that failed ingestion in the batch and retry them a couple times. Actual implementation of this needs to be discusssed.

### Proposed Solution

When bulk request returns successful status but individual items are not successful these items need to be retried - either combined with more items from the queue or just these items alone. If after a couple attempts ingestion fails these items should be dropped with a log line.

### Additional Context
Be mindful of Error Monitor in [_process_bulk_response](https://github.com/elastic/connectors/blob/main/app/connectors_service/connectors/es/sink.py#L238) - it counts failed items and makes a decision to drop some items, this logic should only execute after retrying of the ingestion happened

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in app/connectors_service/connectors/es/sink.py at the bulk collection logic around line 203 and follow _process_bulk_response around line 238. Determine how individual 429 failures are identified and how retries interact with the Error Monitor. Done means failed items are retried a couple of times before the drop-and-log behavior runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.