apache / apache/iggy

Kafka gateway: Kafka gateway IggyBridge core — SDK client and stream/topic mapping

Closed
#3,533 0 comments 0 reactions 0 assignees View on GitHub
gateways
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

Part of #3560.

## Summary

Implement the `IggyBridge` integration layer in `gateways/kafka/` that connects decoded Kafka protocol handlers to the Iggy Rust SDK.

## Motivation

Phase 1 of the [Kafka → Iggy bridge rollout](https://github.com/apache/iggy/discussions/3253#discussioncomment-17181030) requires mapping Kafka topics/partitions to Iggy streams/topics and exposing produce/fetch/watermark operations. This module is shared by all Phase 1 API handlers.

## Depends on

- #3421 — TCP listener, request decode, ApiVersions firewall

## Scope

- [x] `bridge/iggy_bridge.rs` (or equivalent) with async Iggy TCP client lifecycle
- [x] `ensure_stream_and_topic(kafka_topic, partition_count)` → Iggy stream + topic create-if-missing
- [x] Topic name → Iggy stream/topic mapping config (TOML/env)
- [x] Connection config: Iggy address, credentials (`SecretString` pattern)
- [x] `high_watermark(stream, topic, partition)` for ListOffsets
- [x] Error mapping: Iggy errors → Kafka error codes where applicable
- [x] Unit tests with mock or harness; no panics on Iggy unreachable at handler boundary

## Out of scope

- Produce/Fetch handler wiring (separate issues)
- Consumer group offset storage

## Acceptance criteria

- [x] Bridge module compiles and is invoked from integration test stub
- [x] `ensure_stream_and_topic` idempotent on repeated calls
- [ ] README section documents mapping rules and config

## References

- Rollout plan: https://github.com/apache/iggy/discussions/3253#discussioncomment-17181030
- Parent discussion: https://github.com/apache/iggy/discussions/3253
- Related: https://github.com/apache/iggy/discussions/3081
- Prior issue: #3421

Contributor guide

Open the contributing guide

Research direction

Start with gateways/kafka/bridge/iggy_bridge.rs and the integration test stub to verify the implemented mapping and configuration behavior. Check the repository README structure, then document the topic-to-stream/topic mapping rules and TOML or environment configuration; done means the README section is added and the existing tests still pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.