apache / apache/iceberg

Kafka Connect: Idle sink tasks stall every commit round until the timeout

Open
#17,811 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Kafka Connect

### Please describe the bug 🐞

A Kafka Connect sink task that holds partitions but receives no records never
answers the coordinator's commit requests, so every commit round stalls.

### Symptom

- Commit latency is pinned to `iceberg.control.commit.timeout-ms` instead of
completing as soon as the writers report.
- Each round falls back to a partial commit, and partial commits never set the
`kafka.connect.valid-through-ts` snapshot property, so the watermark stops
advancing.

### Root cause

The worker is the task's side of the commit protocol — it reports every assigned
partition on a commit request (with a null offset for partitions that got no
data). That code is correct, but nothing reached it: the worker's lifetime was
tied to record arrival rather than to the partition assignment. `CommitterImpl#save`
created the worker only inside the branch guarded by
`sinkRecords != null && !sinkRecords.isEmpty()`, so an idle task never had a
worker at all.

`close` also stops the worker on every rebalance, and a cooperative rebalance
that only revokes partitions is not followed by `open`, so a task can keep
partitions while its worker stays gone until the next record arrives.

### Proposed fix

Start the worker whenever the task holds an assignment (`ownsPartitions()`),
driven from `save` where `processControlEvents` polls the control topic.

A fix with tests is up in #17593.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with CommitterImpl#save, where processControlEvents polls the control topic, then inspect the worker lifecycle in save, open, close, and ownsPartitions(). Compare the behavior with the tests and fix described in #17593; done means idle assigned partitions answer commit requests and the watermark advances without timeout-driven partial commits.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.