Kafka gateway: Design Consumer offset persistence strategy for Kafka gateway Phase 2
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
Part of #3560.
## Summary
Decide and document how the Kafka gateway stores consumer group offsets for Phase 2.
## Motivation
Open question from https://github.com/apache/iggy/discussions/3253#discussioncomment-17181030:
| Option | Description |
|--------|-------------|
| **A (recommended)** | Iggy-backed `__consumer_offsets` internal topic |
| **B** | SQLite file on bridge host |
| **C** | In-memory only (resets on restart) |
Phase 2 is blocked until this decision is recorded.
## Depends on
- #3539 — Phase 1 complete (or near-complete)
## Deliverables
- [ ] ADR or design comment on #3253 with chosen option
- [ ] Data model: key = `(group, topic, partition)` → committed offset
- [ ] Durability, backup, multi-bridge-instance behavior documented
- [ ] Partition assignment strategy decision: range / round-robin / sticky vs Iggy balanced
## Acceptance criteria
- [ ] Maintainers sign off on Option A/B/C
- [ ] Phase 2 implementation issues unblocked
Contributor guide
Research direction
Start with discussion #3253 and review dependency issue #3539; the payload names no implementation files or tests. Record the selected offset-persistence and partition-assignment strategies, data model, durability, backup, and multi-bridge behavior in an ADR or design comment, with maintainer sign-off unblocking Phase 2.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100