apache / apache/iggy

Discussion: how should sinks handle partial external writes and replay-safe progress?

Open
#2,940 1 comment 0 reactions 0 assignees View on GitHub
connectors design question
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

While reviewing the MongoDB sink follow-up, I ended up at a broader connector-runtime question that likely applies beyond MongoDB.

Some sinks can partially commit a batch in the external system and still return an error. That creates a hard tradeoff between:
- not reporting full success when only part of the batch was written
- not getting stuck replaying already-written messages forever
- keeping progress and restart behavior understandable

A few design questions seem worth discussing more generally:
- should sinks be able to report partial success to the runtime
- should runtime progress be able to advance partially
- should deterministic external write identity be a standard requirement for sinks
- when is duplicate detection enough, and when do we need a stronger contract

I am not proposing a concrete runtime change in this issue yet.

I have a small PR separately to align the MongoDB sink README and test naming with the current merged behavior. This issue is about the larger contract and design question behind partial writes and replay-safe progress.

Contributor guide

Open the contributing guide

Research direction

Start with the MongoDB sink follow-up and the current merged behavior, then trace how connector sinks report errors and how runtime progress behaves across restarts. No file, test, or entry point is named; done would require an agreed contract for partial success, replay-safe progress, deterministic write identity, and duplicate detection.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb, rust
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.