Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
Part of #3560.
## Summary
Implement **OffsetCommit (8)** and **OffsetFetch (9)** with durable storage per #3540 design.
## Motivation
Phase 2 success metrics:
- Offsets persisted across bridge restarts
- `kafka-console-consumer.sh --group` commits and resumes
## Depends on
- #3540 — persistence design
- #3541 — consumer group coordinator
## Scope
- [ ] OffsetCommit: store `(group, topic, partition) → offset`
- [ ] OffsetFetch: return committed offsets on JoinGroup / consumer start
- [ ] Restart test: commit offset, restart bridge, consumer resumes
- [ ] Stub OffsetFetch for unknown group returns empty or appropriate error
## Acceptance criteria
- [ ] Consumer group resumes from last committed offset after bridge restart (Option A or B)
- [ ] At-least-once on failure still documented
## References
- Rollout plan: https://github.com/apache/iggy/discussions/3253#discussioncomment-17181030
Contributor guide
Research direction
Start by reading the persistence design in #3540 and the consumer group coordinator work in #3541, then trace the Kafka gateway entry points for OffsetCommit (8) and OffsetFetch (9). Verify the restart scenario by committing an offset, restarting the bridge, and confirming the consumer resumes; also cover unknown groups and document at-least-once behavior on failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, rust
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100