apache / apache/iceberg

Control topic always increasing

Open
#15,844 2 comments 1 reaction 0 assignees View on GitHub
question
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Query engine

- AWS MSK kafka connect v 3.7.x and iceberg sink 1.10.1
- using Glue catalog, writing to S3

### Question

I have experimented with different settings of `iceberg.control.commit.interval-ms` and `iceberg.control.commit.timeout-ms`. Interval is usually set to 5 minutes, and timeout from 2 minutes up to 8.
I even tried increasing `iceberg.kafka.session.timeout.ms` and `iceberg.kafka.heartbeat.interval.ms`.

Image

In all cases I see the control topic growing over time with this kind of a pattern:

Image

I have observed these in logs:
- Received commit response when no commit in progress, this can happen during recovery.
- Commit timeout reached.

I do not see any other errors, data is flowing as expected, topics do get drained. Except the control one, which will blow up one day since it is constantly increasing. There are 5 iceberg sinks, all using the same control topic.

I have noticed that if I destroy the connector and re-create it after a short while, the control topic "will start growing from 0 again". Or maybe I was just lucky!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.