Kafka Streams runner: finalize unbounded source checkpoints
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The Kafka Streams runner stores an unbounded source's checkpoint mark so a restart resumes from it, but never calls `CheckpointMark.finalizeCheckpoint()`. Sources that rely on finalization to acknowledge or release data will not see it.
A mark should only be finalized once it is durably committed, which needs a hook that runs before Kafka Streams commits. The runner does not have one — the same gap that prevents bundles being bounded by time. A related point from review: a bounded split should likewise only be committed after it has been completely processed and flushed downstream.
Part of #18479.
Contributor guide
Research direction
Start in the Kafka Streams runner at the checkpoint persistence and Kafka Streams commit path, then trace CheckpointMark.finalizeCheckpoint(). Done means unbounded source marks are finalized only after durable commit, and bounded splits are committed only after complete processing and downstream flush.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100