apache / apache/iggy

Kafka gateway: protocol parity tracking

Open
#3,560 0 comments 0 reactions 0 assignees View on GitHub
gateways
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

Umbrella tracking issue for the Kafka-wire-protocol gateway (`gateways/`), letting existing Kafka clients talk to Iggy unmodified.

Seed design + TCP listener lands in #3421.

Goal: implement enough of the Kafka wire protocol that standard Kafka producers/consumers (and admin tools) work against Iggy via a bridge, then document migration off Kafka.

## Sub-issues

Foundation:

- [x] #3421 - TCP listener for Kafka clients (seed)
- [ ] #3533 - IggyBridge core: SDK client + stream/topic mapping

Phase 1 - core wire APIs + tooling:

- [ ] #3534 - Metadata API (key 3): broker/topic discovery
- [ ] #3535 - Produce API (key 0): bridge to `send_messages`
- [ ] #3536 - Fetch API (key 1): bridge to offset-based polling
- [ ] #3537 - ListOffsets API (key 2): earliest/latest via high watermark
- [ ] #3538 - CreateTopics API (key 19): stream/topic provisioning
- [ ] #3539 - Phase 1: docker-compose, E2E tests, quick-start docs

Phase 2 - consumer groups + offsets:

- [ ] #3540 - design: consumer offset persistence strategy
- [ ] #3541 - FindCoordinator, JoinGroup, SyncGroup, Heartbeat
- [ ] #3542 - OffsetCommit + OffsetFetch with persisted offsets
- [ ] #3543 - LeaveGroup + rebalance semantics
- [ ] #3544 - Phase 2: E2E tests with consumer groups

Admin + extended APIs:

- [ ] #3545 - transactional APIs: UNSUPPORTED_VERSION stub + idempotence docs
- [ ] #3546 - DeleteTopics, DescribeConfigs, AlterConfigs
- [ ] #3547 - DeleteRecords: trim topic to offset via retention
- [ ] #3548 - DescribeGroups + ListGroups

Auth + migration:

- [ ] #3549 - Kafka SASL auth bridge to Iggy users
- [ ] #3550 - migration guide: dual-write, cutover, decommission

## Status

Stays open until boxes above are checked or de-scoped.

Contributor guide

Open the contributing guide

Research direction

This is an umbrella issue for the Kafka-wire-protocol work in gateways/, with the seed TCP listener tracked in #3421. Start by reading #3421 and selecting one unchecked sub-issue that defines a concrete API or documentation task. The umbrella is done when the listed work is checked off or explicitly de-scoped.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, rust
Domain
api, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.