apache / apache/fluss

[kafka] Add request and transport framework

Open
#4,264 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Parent issue

Part of #4185 — Kafka Protocols Compatibility: Basic Produce.

### Motivation

Provide the reusable request and transport foundation for small, independently reviewable Kafka API implementations. This issue establishes the infrastructure; it does not implement a new concrete API handler.

### Scope

- Add request context, API specifications, typed handlers, a capability registry, asynchronous dispatch, and common error mapping.
- Propagate listener and connection context from the Kafka transport.
- Handle request/response buffer ownership, including idempotent release and cleanup on serialization failure.
- Recognize configured Kafka protocol endpoints in the Netty server.
- Retain the existing request-handler behavior until the ApiVersions follow-up connects the dispatcher to production.

### Acceptance criteria

- [ ] Registry and dispatcher tests cover registration, version checks, asynchronous completion, and handler failures.
- [ ] Request context reaches handlers without losing listener or connection information.
- [ ] Request and response buffers are released correctly on successful and failing paths.
- [ ] Relevant Kafka and RPC unit/integration tests, formatting, and license checks pass.
- [ ] The focused implementation is reviewed and merged before dependent stages are completed.

### Out of scope

Concrete ApiVersions/Metadata handlers, Produce, authentication, schema conversion, Kafka CreateTopics/DeleteTopics, and automatic table creation.

### Implementation and review

Implementation PR: #4259.

This is the first foundation stage. The ApiVersions and Metadata issues build on it. Track completion in this issue rather than in a PR checklist on the umbrella.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with implementation PR #4259, then inspect the Kafka transport and Netty server areas named in the scope. Use the registry and dispatcher tests as the main entry point, and verify the listed acceptance criteria, including context propagation, buffer cleanup, asynchronous completion, and relevant Kafka/RPC checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.