Kafka Connect: Idle sink tasks stall every commit round until the timeout
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Kafka Connect
### Please describe the bug 🐞
A Kafka Connect sink task that holds partitions but receives no records never
answers the coordinator's commit requests, so every commit round stalls.
### Symptom
- Commit latency is pinned to `iceberg.control.commit.timeout-ms` instead of
completing as soon as the writers report.
- Each round falls back to a partial commit, and partial commits never set the
`kafka.connect.valid-through-ts` snapshot property, so the watermark stops
advancing.
### Root cause
The worker is the task's side of the commit protocol — it reports every assigned
partition on a commit request (with a null offset for partitions that got no
data). That code is correct, but nothing reached it: the worker's lifetime was
tied to record arrival rather than to the partition assignment. `CommitterImpl#save`
created the worker only inside the branch guarded by
`sinkRecords != null && !sinkRecords.isEmpty()`, so an idle task never had a
worker at all.
`close` also stops the worker on every rebalance, and a cooperative rebalance
that only revokes partitions is not followed by `open`, so a task can keep
partitions while its worker stays gone until the next record arrives.
### Proposed fix
Start the worker whenever the task holds an assignment (`ownsPartitions()`),
driven from `save` where `processControlEvents` polls the control topic.
A fix with tests is up in #17593.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with CommitterImpl#save, where processControlEvents polls the control topic, then inspect the worker lifecycle in save, open, close, and ownsPartitions(). Compare the behavior with the tests and fix described in #17593; done means idle assigned partitions answer commit requests and the watermark advances without timeout-driven partial commits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 30/100