apache / apache/iggy

connectors: fix producer.send() failure lets a later batch supersede the skipped state save

Open
#3,898 1 comment 0 reactions 0 assignees View on GitHub
connectors
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

Split out of the #3795 review at hubcio's suggestion, so it is not lost when that PR merges.

## The shape

`source_forwarding_loop` persists a source's `ConnectorState` only on the success branch of the Iggy send. When `producer.send()` fails, the loop logs, skips the save, and continues to the next batch.

Sources advance their cursor at poll time and snapshot it into every batch (`tracking_offsets` in postgres_source, same shape in the others). So the next batch that *does* land persists a cursor that already covers the batch whose send failed, and a restart resumes past it.

That is the same supersession #3795 fixes for the shutdown-drop path with a per-instance `dropped` latch. The latch does not close this one: it is armed only by a drop, not by a send failure.

## Why it is filed separately rather than fixed there

Two differences from the shutdown case, both of which argue for treating it on its own:

- It predates #3795 and exists on master today.
- It flips the connector to `Error` status, so unlike the shutdown drop it is at least visible, rather than only logged and counted.

Whether the fix is the same latch armed from this path, a retry, or something that surfaces the gap rather than hiding it, is a design question worth its own discussion.

Happy to take it once the direction is settled.

Contributor guide

Open the contributing guide

Research direction

Start at source_forwarding_loop and compare its failed-send path with the per-instance dropped latch from #3795. Trace tracking_offsets in postgres_source and the equivalent source batches to decide how a failed send must affect ConnectorState persistence. Done means a failed batch cannot be superseded by a later state save and a restart does not resume past it; the chosen behavior should have regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.