Kafka-connect : many commit Timeout in a row never fail the connector
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Apache Iceberg version
1.7.1
### Query engine
Kafka Connect
### Please describe the bug 🐞
My connector was failing to commit during multiples hours , I detected it thanks to the consumer group lag
but the connector never failed , so it look like there is not a max-retry policy on the commit timeout
Would be great to have this and be able to custom this value
Thanks all
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by locating the Kafka Connect commit handling and its timeout retry behavior. Reproduce or trace consecutive commit timeouts, then verify that a configurable maximum retry policy causes the connector to fail instead of continuing indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100