apache / apache/iceberg

Kafka-connect : many commit Timeout in a row never fail the connector

Open
#13,117 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

1.7.1

### Query engine

Kafka Connect

### Please describe the bug 🐞

My connector was failing to commit during multiples hours , I detected it thanks to the consumer group lag

but the connector never failed , so it look like there is not a max-retry policy on the commit timeout

Would be great to have this and be able to custom this value

Thanks all

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by locating the Kafka Connect commit handling and its timeout retry behavior. Reproduce or trace consecutive commit timeouts, then verify that a configurable maximum retry policy causes the connector to fail instead of continuing indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.