Kafka Connect: `cg-control` consumer group fails to stabilize on idle/low-volume topics after PR #14395
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Kafka Connect
### Please describe the bug 🐞
### Summary
After upgrading from Iceberg 1.10.1 to 1.11.0, the Kafka Connect Iceberg Sink connector's `cg-control` consumer group cannot reach STABLE state when the data topic goes idle after initial consumption. This causes all commits to timeout with "committed to 0 table(s)".
The regression was introduced by PR #14395 ("Don't check that consumer group is stable for coordinator leader election"), which removed the stability check that previously ensured the Worker's `cg-control` consumer had joined before the Coordinator started firing commits.
### Root Cause
In 1.10.1, the Coordinator waited for the `cg-control` consumer group to be STABLE before starting commit cycles. In 1.11.0, this check was removed. The unintended consequence:
1. Connector starts → consumes all available messages in the first `poll()` call
2. Data topic now has no new messages → next `poll()` blocks for `max.poll.interval.ms` (default 60s)
3. During this 60s block, the `cg-control` consumer's `poll()` is not called (single-threaded task loop)
4. `cg-control` session timeout is 45s (default `session.timeout.ms`)
5. After 45s with no heartbeat → broker kicks `cg-control` member → `UnknownMemberIdException`
6. `cg-control` can never reach STABLE → Worker never processes START_COMMIT → commit always times out
The 45s session timeout < 60s poll block creates a guaranteed failure condition whenever the data topic goes idle.
### Steps to Reproduce
1. Deploy Iceberg Sink Connector 1.11.0 in distributed mode with `tasks.max=1`
2. Have a data topic with a finite set of messages (e.g., a CDC snapshot — no continuous flow)
3. Configure `iceberg.control.commit.interval-ms: 60000` (or any value > session timeout)
4. Start the connector from a clean state (no internal topics)
5. Connector consumes all messages in first poll
6. Topic goes idle → `cg-control` consumer can never join → all commits timeout with 0 tables
### Expected Behavior
The `cg-control` consumer group should reach STABLE state regardless of data topic activity. Either:
- The Coordinator should wait for `cg-control` stability before firing the first commit (as in 1.10.1)
- Or the `cg-control` consumer's `poll()` should be decoupled from the data consumer's poll loop (separate thread)
### Actual Behavior
```
[TRACE] Polling returned 454 messages ← All consumed in first poll
[TRACE] Delivering batch of 454 messages ← Delivered to put()
... 60s passes (data poll blocks) ...
[INFO] Commit timeout reached → committed to 0 table(s)
[INFO] [cg-control-*] Request joining group due to: rebalance failed due to
'The coordinator is not aware of this member.' (UnknownMemberIdException)
```
The `cg-control` consumer repeatedly fails to join:
```
sessionTimeoutMs=45000 ← Broker kicks after 45s
rebalanceTimeoutMs=300000
```
But the main task loop only iterates every 60s (poll timeout), so `cg-control` gets only one `poll()` call per 60s — insufficient to maintain the 45s session.
### Comparison: 1.10.1 vs 1.11.0
| Version | `cg-control` State | Commits Work? | Behavior |
|---------|-------------------|---------------|----------|
| 1.10.1 | STABLE ✅ | Yes | Coordinator waits for stability |
| 1.11.0 | EMPTY ❌ | No | Coordinator fires immediately |
Tested with identical connector config on the same Kafka cluster. The only difference is the Iceberg plugin version.
### Workaround
Setting `iceberg.kafka.session.timeout.ms` to a value greater than the data consumer's poll timeout:
```json
"iceberg.kafka.session.timeout.ms": "120000",
"iceberg.kafka.heartbeat.interval.ms": "10000"
```
This passes through to the `cg-control` consumer via the `iceberg.kafka.*` prefix, preventing the session from expiring during the idle poll block.
Alternatively, ensuring continuous data flow keeps the poll loop short, allowing `cg-control` to join — but this is not a viable solution for batch/snapshot workloads.
### Related Issues
- [#11818 ](https://github.com/apache/iceberg/issues/11818)— "Iceberg Kafka Connector experiences a constant hanging lag for low-volume topics" (same symptom, reported on 1.7.1, closed as stale)
- [#14395](https://github.com/apache/iceberg/pull/14395)— The PR that introduced this regression
### Environment
- Kafka Connect: 3.9.2 (distributed mode)
- Kafka Broker: Amazon MSK 3.9.2
- Iceberg Sink: 1.11.0 (built from `apache-iceberg-1.11.0` tag)
- `tasks.max`: 1
- `iceberg.control.commit.interval-ms`: 60000
- Data topic: 454 messages (CDC snapshot, no continuous flow)
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
## Related problems
- https://github.com/apache/iceberg/issues/12608 Some how I feel can it be the reason too ? and this ticket there is no conclusion
Contributor guide
Research direction
Start by reviewing PR #14395 and comparing the Coordinator behavior between Iceberg 1.10.1 and 1.11.0. Reproduce the failure with Kafka Connect 3.9.2, tasks.max=1, a finite topic, and a 60-second commit interval. Done means the cg-control group remains stable and commits complete after the data topic becomes idle; the issue leaves the implementation choice open.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- backend, data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100