Enable Kafka coordinator fencing when committing consumer offsets
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
### Description
Add support for coordinator fencing. This can prevent stale ("zombie") coordinators from committing stale consumer offsets.
### Implementation
Currently, we generate a new UUID for every new coordinator. This means that the Kafka transaction we use trivially passes since no epoch fencing can be performed on the newly generated ID (e.g., between a stale or an up to date coordinator).
The change is to use a persistent transactional ID for the coordinator. Thus, any leader change of the coordinator group will trigger Kafka to update the epoch of the transaction, and correctly fence stale coordinators. A stale coordinator will identify that it has been fenced, and exit the coordinator thread. We leave the worker task running and defer to Kafka Connect on determining its lifecycle.
See [PR](https://github.com/apache/iceberg/pull/18039) for more details
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start by reading PR 18039 and the Kafka coordinator transaction and coordinator-thread entry points it describes. Verify how a persistent transactional ID changes fencing on coordinator leadership changes, how a fenced coordinator exits, and that the worker task remains running; done means stale coordinators can no longer commit offsets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100