apache / apache/iggy

Kafka gateway: Design Consumer offset persistence strategy for Kafka gateway Phase 2

Open
#3,540 0 comments 0 reactions 0 assignees View on GitHub
gateways
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

Part of #3560.

## Summary

Decide and document how the Kafka gateway stores consumer group offsets for Phase 2.

## Motivation

Open question from https://github.com/apache/iggy/discussions/3253#discussioncomment-17181030:

| Option | Description |
|--------|-------------|
| **A (recommended)** | Iggy-backed `__consumer_offsets` internal topic |
| **B** | SQLite file on bridge host |
| **C** | In-memory only (resets on restart) |

Phase 2 is blocked until this decision is recorded.

## Depends on

- #3539 — Phase 1 complete (or near-complete)

## Deliverables

- [ ] ADR or design comment on #3253 with chosen option
- [ ] Data model: key = `(group, topic, partition)` → committed offset
- [ ] Durability, backup, multi-bridge-instance behavior documented
- [ ] Partition assignment strategy decision: range / round-robin / sticky vs Iggy balanced

## Acceptance criteria

- [ ] Maintainers sign off on Option A/B/C
- [ ] Phase 2 implementation issues unblocked

Contributor guide

Open the contributing guide

Research direction

Start with discussion #3253 and review dependency issue #3539; the payload names no implementation files or tests. Record the selected offset-persistence and partition-assignment strategies, data model, durability, backup, and multi-bridge behavior in an ADR or design comment, with maintainer sign-off unblocking Phase 2.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.