KafkaConnect: Add JMX metrics for Coordinator/Worker
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
The current Kafka-connect-iceberg component did not add any new JMX metrics, we can only use the existing metrics from Kafka consumer and Kafka Connect framework. When we were diagnosing performance problems related to slow or paused commit from Coordinator, we have to turn on many low level DEBUG/TRACE logging category to sense the system bottleneck.
A well designed framework should have its own metrics reporting system for its key operational components for monitoring and diagnostic purposes. I think the following metrics would be useful to report:
For Coordinator: commit-time, consume-available-time, commit-buffer size, ready-buffer size, message counts for START_COMMIT and COMMIT_COMPLETE messages.
For Worker: save-time, consume-available-time, message counts for DATA_WRITTEN and DATA_COMPLETE messages.
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Locate the Kafka-connect-iceberg Coordinator and Worker implementations and their existing metrics or reporting hooks. Trace the commit, consume, buffer, and message-count paths listed in the issue, then verify that each requested metric is exposed for monitoring and diagnostics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100