Discussion: how should sinks handle partial external writes and replay-safe progress?
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
While reviewing the MongoDB sink follow-up, I ended up at a broader connector-runtime question that likely applies beyond MongoDB.
Some sinks can partially commit a batch in the external system and still return an error. That creates a hard tradeoff between:
- not reporting full success when only part of the batch was written
- not getting stuck replaying already-written messages forever
- keeping progress and restart behavior understandable
A few design questions seem worth discussing more generally:
- should sinks be able to report partial success to the runtime
- should runtime progress be able to advance partially
- should deterministic external write identity be a standard requirement for sinks
- when is duplicate detection enough, and when do we need a stronger contract
I am not proposing a concrete runtime change in this issue yet.
I have a small PR separately to align the MongoDB sink README and test naming with the current merged behavior. This issue is about the larger contract and design question behind partial writes and replay-safe progress.
Contributor guide
Research direction
Start with the MongoDB sink follow-up and the current merged behavior, then trace how connector sinks report errors and how runtime progress behaves across restarts. No file, test, or entry point is named; done would require an agreed contract for partial success, replay-safe progress, deterministic write identity, and duplicate detection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb, rust
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100