[kafka] Add request and transport framework
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Parent issue
Part of #4185 — Kafka Protocols Compatibility: Basic Produce.
### Motivation
Provide the reusable request and transport foundation for small, independently reviewable Kafka API implementations. This issue establishes the infrastructure; it does not implement a new concrete API handler.
### Scope
- Add request context, API specifications, typed handlers, a capability registry, asynchronous dispatch, and common error mapping.
- Propagate listener and connection context from the Kafka transport.
- Handle request/response buffer ownership, including idempotent release and cleanup on serialization failure.
- Recognize configured Kafka protocol endpoints in the Netty server.
- Retain the existing request-handler behavior until the ApiVersions follow-up connects the dispatcher to production.
### Acceptance criteria
- [ ] Registry and dispatcher tests cover registration, version checks, asynchronous completion, and handler failures.
- [ ] Request context reaches handlers without losing listener or connection information.
- [ ] Request and response buffers are released correctly on successful and failing paths.
- [ ] Relevant Kafka and RPC unit/integration tests, formatting, and license checks pass.
- [ ] The focused implementation is reviewed and merged before dependent stages are completed.
### Out of scope
Concrete ApiVersions/Metadata handlers, Produce, authentication, schema conversion, Kafka CreateTopics/DeleteTopics, and automatic table creation.
### Implementation and review
Implementation PR: #4259.
This is the first foundation stage. The ApiVersions and Metadata issues build on it. Track completion in this issue rather than in a PR checklist on the umbrella.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with implementation PR #4259, then inspect the Kafka transport and Netty server areas named in the scope. Use the registry and dispatcher tests as the main entry point, and verify the listed acceptance criteria, including context propagation, buffer cleanup, asynchronous completion, and relevant Kafka/RPC checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100