Kafka gateway: protocol parity tracking
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
Umbrella tracking issue for the Kafka-wire-protocol gateway (`gateways/`), letting existing Kafka clients talk to Iggy unmodified.
Seed design + TCP listener lands in #3421.
Goal: implement enough of the Kafka wire protocol that standard Kafka producers/consumers (and admin tools) work against Iggy via a bridge, then document migration off Kafka.
## Sub-issues
Foundation:
- [x] #3421 - TCP listener for Kafka clients (seed)
- [ ] #3533 - IggyBridge core: SDK client + stream/topic mapping
Phase 1 - core wire APIs + tooling:
- [ ] #3534 - Metadata API (key 3): broker/topic discovery
- [ ] #3535 - Produce API (key 0): bridge to `send_messages`
- [ ] #3536 - Fetch API (key 1): bridge to offset-based polling
- [ ] #3537 - ListOffsets API (key 2): earliest/latest via high watermark
- [ ] #3538 - CreateTopics API (key 19): stream/topic provisioning
- [ ] #3539 - Phase 1: docker-compose, E2E tests, quick-start docs
Phase 2 - consumer groups + offsets:
- [ ] #3540 - design: consumer offset persistence strategy
- [ ] #3541 - FindCoordinator, JoinGroup, SyncGroup, Heartbeat
- [ ] #3542 - OffsetCommit + OffsetFetch with persisted offsets
- [ ] #3543 - LeaveGroup + rebalance semantics
- [ ] #3544 - Phase 2: E2E tests with consumer groups
Admin + extended APIs:
- [ ] #3545 - transactional APIs: UNSUPPORTED_VERSION stub + idempotence docs
- [ ] #3546 - DeleteTopics, DescribeConfigs, AlterConfigs
- [ ] #3547 - DeleteRecords: trim topic to offset via retention
- [ ] #3548 - DescribeGroups + ListGroups
Auth + migration:
- [ ] #3549 - Kafka SASL auth bridge to Iggy users
- [ ] #3550 - migration guide: dual-write, cutover, decommission
## Status
Stays open until boxes above are checked or de-scoped.
Contributor guide
Research direction
This is an umbrella issue for the Kafka-wire-protocol work in gateways/, with the seed TCP listener tracked in #3421. Start by reading #3421 and selecting one unchecked sub-issue that defines a concrete API or documentation task. The umbrella is done when the listed work is checked off or explicitly de-scoped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, rust
- Domain
- api, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100