connectors: fix producer.send() failure lets a later batch supersede the skipped state save
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
Split out of the #3795 review at hubcio's suggestion, so it is not lost when that PR merges.
## The shape
`source_forwarding_loop` persists a source's `ConnectorState` only on the success branch of the Iggy send. When `producer.send()` fails, the loop logs, skips the save, and continues to the next batch.
Sources advance their cursor at poll time and snapshot it into every batch (`tracking_offsets` in postgres_source, same shape in the others). So the next batch that *does* land persists a cursor that already covers the batch whose send failed, and a restart resumes past it.
That is the same supersession #3795 fixes for the shutdown-drop path with a per-instance `dropped` latch. The latch does not close this one: it is armed only by a drop, not by a send failure.
## Why it is filed separately rather than fixed there
Two differences from the shutdown case, both of which argue for treating it on its own:
- It predates #3795 and exists on master today.
- It flips the connector to `Error` status, so unlike the shutdown drop it is at least visible, rather than only logged and counted.
Whether the fix is the same latch armed from this path, a retry, or something that surfaces the gap rather than hiding it, is a design question worth its own discussion.
Happy to take it once the direction is settled.
Contributor guide
Research direction
Start at source_forwarding_loop and compare its failed-send path with the per-instance dropped latch from #3795. Trace tracking_offsets in postgres_source and the equivalent source batches to decide how a failed send must affect ConnectorState persistence. Done means a failed batch cannot be superseded by a later state save and a restart does not resume past it; the chosen behavior should have regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100