apache / apache/iceberg

Kafka Connect: `cg-control` consumer group fails to stabilize on idle/low-volume topics after PR #14395

Open
#17,193 1 comment 1 reaction 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Kafka Connect

### Please describe the bug 🐞

### Summary

After upgrading from Iceberg 1.10.1 to 1.11.0, the Kafka Connect Iceberg Sink connector's `cg-control` consumer group cannot reach STABLE state when the data topic goes idle after initial consumption. This causes all commits to timeout with "committed to 0 table(s)".

The regression was introduced by PR #14395 ("Don't check that consumer group is stable for coordinator leader election"), which removed the stability check that previously ensured the Worker's `cg-control` consumer had joined before the Coordinator started firing commits.

### Root Cause

In 1.10.1, the Coordinator waited for the `cg-control` consumer group to be STABLE before starting commit cycles. In 1.11.0, this check was removed. The unintended consequence:

1. Connector starts → consumes all available messages in the first `poll()` call
2. Data topic now has no new messages → next `poll()` blocks for `max.poll.interval.ms` (default 60s)
3. During this 60s block, the `cg-control` consumer's `poll()` is not called (single-threaded task loop)
4. `cg-control` session timeout is 45s (default `session.timeout.ms`)
5. After 45s with no heartbeat → broker kicks `cg-control` member → `UnknownMemberIdException`
6. `cg-control` can never reach STABLE → Worker never processes START_COMMIT → commit always times out

The 45s session timeout < 60s poll block creates a guaranteed failure condition whenever the data topic goes idle.

### Steps to Reproduce

1. Deploy Iceberg Sink Connector 1.11.0 in distributed mode with `tasks.max=1`
2. Have a data topic with a finite set of messages (e.g., a CDC snapshot — no continuous flow)
3. Configure `iceberg.control.commit.interval-ms: 60000` (or any value > session timeout)
4. Start the connector from a clean state (no internal topics)
5. Connector consumes all messages in first poll
6. Topic goes idle → `cg-control` consumer can never join → all commits timeout with 0 tables

### Expected Behavior

The `cg-control` consumer group should reach STABLE state regardless of data topic activity. Either:
- The Coordinator should wait for `cg-control` stability before firing the first commit (as in 1.10.1)
- Or the `cg-control` consumer's `poll()` should be decoupled from the data consumer's poll loop (separate thread)

### Actual Behavior

```
[TRACE] Polling returned 454 messages ← All consumed in first poll
[TRACE] Delivering batch of 454 messages ← Delivered to put()
... 60s passes (data poll blocks) ...
[INFO] Commit timeout reached → committed to 0 table(s)
[INFO] [cg-control-*] Request joining group due to: rebalance failed due to
'The coordinator is not aware of this member.' (UnknownMemberIdException)
```

The `cg-control` consumer repeatedly fails to join:
```
sessionTimeoutMs=45000 ← Broker kicks after 45s
rebalanceTimeoutMs=300000
```

But the main task loop only iterates every 60s (poll timeout), so `cg-control` gets only one `poll()` call per 60s — insufficient to maintain the 45s session.

### Comparison: 1.10.1 vs 1.11.0

| Version | `cg-control` State | Commits Work? | Behavior |
|---------|-------------------|---------------|----------|
| 1.10.1 | STABLE ✅ | Yes | Coordinator waits for stability |
| 1.11.0 | EMPTY ❌ | No | Coordinator fires immediately |

Tested with identical connector config on the same Kafka cluster. The only difference is the Iceberg plugin version.

### Workaround

Setting `iceberg.kafka.session.timeout.ms` to a value greater than the data consumer's poll timeout:

```json
"iceberg.kafka.session.timeout.ms": "120000",
"iceberg.kafka.heartbeat.interval.ms": "10000"
```

This passes through to the `cg-control` consumer via the `iceberg.kafka.*` prefix, preventing the session from expiring during the idle poll block.

Alternatively, ensuring continuous data flow keeps the poll loop short, allowing `cg-control` to join — but this is not a viable solution for batch/snapshot workloads.

### Related Issues

- [#11818 ](https://github.com/apache/iceberg/issues/11818)— "Iceberg Kafka Connector experiences a constant hanging lag for low-volume topics" (same symptom, reported on 1.7.1, closed as stale)
- [#14395](https://github.com/apache/iceberg/pull/14395)— The PR that introduced this regression

### Environment

- Kafka Connect: 3.9.2 (distributed mode)
- Kafka Broker: Amazon MSK 3.9.2
- Iceberg Sink: 1.11.0 (built from `apache-iceberg-1.11.0` tag)
- `tasks.max`: 1
- `iceberg.control.commit.interval-ms`: 60000
- Data topic: 454 messages (CDC snapshot, no continuous flow)

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

## Related problems
- https://github.com/apache/iceberg/issues/12608 Some how I feel can it be the reason too ? and this ticket there is no conclusion

Contributor guide

Open the contributing guide

Research direction

Start by reviewing PR #14395 and comparing the Coordinator behavior between Iceberg 1.10.1 and 1.11.0. Reproduce the failure with Kafka Connect 3.9.2, tasks.max=1, a finite topic, and a 60-second commit interval. Done means the cg-control group remains stable and commits complete after the data topic becomes idle; the issue leaves the implementation choice open.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
backend, data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.