apache / apache/iggy

Kafka gateway: Kafka OffsetCommit and OffsetFetch with persisted consumer offsets

Open
#3,542 0 comments 0 reactions 0 assignees View on GitHub
gateways
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

Part of #3560.

## Summary

Implement **OffsetCommit (8)** and **OffsetFetch (9)** with durable storage per #3540 design.

## Motivation

Phase 2 success metrics:
- Offsets persisted across bridge restarts
- `kafka-console-consumer.sh --group` commits and resumes

## Depends on

- #3540 — persistence design
- #3541 — consumer group coordinator

## Scope

- [ ] OffsetCommit: store `(group, topic, partition) → offset`
- [ ] OffsetFetch: return committed offsets on JoinGroup / consumer start
- [ ] Restart test: commit offset, restart bridge, consumer resumes
- [ ] Stub OffsetFetch for unknown group returns empty or appropriate error

## Acceptance criteria

- [ ] Consumer group resumes from last committed offset after bridge restart (Option A or B)
- [ ] At-least-once on failure still documented

## References

- Rollout plan: https://github.com/apache/iggy/discussions/3253#discussioncomment-17181030

Contributor guide

Open the contributing guide

Research direction

Start by reading the persistence design in #3540 and the consumer group coordinator work in #3541, then trace the Kafka gateway entry points for OffsetCommit (8) and OffsetFetch (9). Verify the restart scenario by committing an offset, restarting the bridge, and confirming the consumer resumes; also cover unknown groups and document at-least-once behavior on failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, rust
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.