connectors: preserve failed records throughout the sink pipeline
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- rust
- Domain
- stream-processing
Research direction
Start by tracing the connector sink pipeline through decoding, transformation, serialization, and delivery, then read the accounting introduced by #4152 alongside #4233 and #4234. Done means stage failures, filtering, plugin retries, and dead-letter delivery remain distinguishable, while checkpoints and destination results preserve unresolved records correctly; tests must cover failures at each stage.
Written by the indexing model from the issue text.
Description
Parent tracker: #4232.
Carry record outcomes through decoding, transformation, serialization, and sink delivery so that a successful call for the remaining records cannot silently acknowledge records dropped earlier in the pipeline.
Acceptance criteria:
- Unexpected failures leave the affected source records unresolved and block checkpoints from passing them.
- Configured filtering is represented and counted separately from errors.
- Error information crosses the plugin boundary without collapsing every failure into an indistinguishable status.
- Retry ownership and limits are explicit, including retries already performed by plugins. Exhaustion pauses or stops affected work with an actionable error and preserves replay.
- Failure handling bounds pending work and does not abandon records already covered by a checkpoint.
- Any supported dead-letter policy confirms delivery to that destination before resolving the source record. Without such a policy, permanent record failures remain visible and unresolved.
- Tests exercise failures at each stage and verify the saved checkpoint and destination results.
Depends on #4233 and uses #4234. Extend the accounting introduced by #4152 while preserving the original input record's outcome.
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 442
- Avg merge
- 2d 23m
- Merged PRs (30d)
- 153
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/iggy
-
server test
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
connectors: runtime discards the sink consume FFI status, so failed batches are counted as processed Openbug connectors rust
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
bug python sdk
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 5/5 Over a week Newbie friendliness 38/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
kwakseongjae/auto-hwp#319 ·
-
area:cli bug filter-quality good first issue priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
bevyengine/bevy#25861 ·
-
comp-datalake
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
ClickHouse/ClickHouse#121222 ·
-
enhancement remote
Difficulty 2/5 1-3 hours Newbie friendliness 68/100