apache / apache/iceberg

KafkaConnect: Add JMX metrics for Coordinator/Worker

Open
#17,026 1 comment 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

The current Kafka-connect-iceberg component did not add any new JMX metrics, we can only use the existing metrics from Kafka consumer and Kafka Connect framework. When we were diagnosing performance problems related to slow or paused commit from Coordinator, we have to turn on many low level DEBUG/TRACE logging category to sense the system bottleneck.

A well designed framework should have its own metrics reporting system for its key operational components for monitoring and diagnostic purposes. I think the following metrics would be useful to report:

For Coordinator: commit-time, consume-available-time, commit-buffer size, ready-buffer size, message counts for START_COMMIT and COMMIT_COMPLETE messages.

For Worker: save-time, consume-available-time, message counts for DATA_WRITTEN and DATA_COMPLETE messages.

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Locate the Kafka-connect-iceberg Coordinator and Worker implementations and their existing metrics or reporting hooks. Trace the commit, consume, buffer, and message-count paths listed in the issue, then verify that each requested metric is exposed for monitoring and diagnostics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.