Control topic always increasing
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Query engine
- AWS MSK kafka connect v 3.7.x and iceberg sink 1.10.1
- using Glue catalog, writing to S3
### Question
I have experimented with different settings of `iceberg.control.commit.interval-ms` and `iceberg.control.commit.timeout-ms`. Interval is usually set to 5 minutes, and timeout from 2 minutes up to 8.
I even tried increasing `iceberg.kafka.session.timeout.ms` and `iceberg.kafka.heartbeat.interval.ms`.
In all cases I see the control topic growing over time with this kind of a pattern:
I have observed these in logs:
- Received commit response when no commit in progress, this can happen during recovery.
- Commit timeout reached.
I do not see any other errors, data is flowing as expected, topics do get drained. Except the control one, which will blow up one day since it is constantly increasing. There are 5 iceberg sinks, all using the same control topic.
I have noticed that if I destroy the connector and re-create it after a short while, the control topic "will start growing from 0 again". Or maybe I was just lucky!
Contributor guide
Assessment
This issue has not been assessed yet.