[Umbrella] Kafka Protocols Compatibility - Basic Produce
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
Apache Fluss needs a first Kafka protocol compatibility milestone that allows standard Kafka producers to discover Fluss tables and write records without changing the Kafka client.
The target workflow is **DDL-first**: users create and manage tables and schemas through Fluss DDL, then produce JSON records through the Kafka protocol. Schema-aware conversion maps records into the pre-defined table schema and existing Fluss storage format. Kafka `CreateTopics`, `DeleteTopics`, and automatic table creation are not part of this milestone.
This umbrella is issue-driven: define a focused child issue, link its implementation PR, review it, and then move to the next capability. The foundation and Basic Produce tasks below follow dependency order. The previous four-PR approach has been retired and is not part of the current dependency chain.
### Sub-tasks
Only scoped child issues are listed here. Each issue owns its scope, acceptance criteria, dependencies, and implementation PR. An issue is complete after the reviewed implementation meets those criteria and is merged, not merely after a PR is opened.
**Foundation**
- [ ] #4264 — PR01: Request and transport framework.
- [ ] #4265 — PR02: Registry-backed ApiVersions API.
- [ ] #4275 — PR03: DDL table mapping contract.
- [ ] #4266 — PR04: Metadata API for pre-created Fluss tables.
Review and dependency order: #4264 → #4265 → #4275 → #4266.
**Basic Produce**
- [ ] #4281 — PR05: Produce protocol handling.
- [ ] #4282 — PR06: Raw/string record conversion.
- [ ] #4283 — PR07: Native append integration.
- [ ] #4298 — PR08: Produce lifecycle and replication regressions.
**Core changes (independent review and merge)**
- [ ] #3455 — Complete delayed fetch after produce.
### Planned follow-ups
Create additional child issues progressively after the scoped foundation and Basic Produce tasks. These are directions for later issue scoping, not a commitment to reuse the closed implementation PRs.
**Capability**
- SASL/PLAIN authentication: handshake, per-connection authentication state, and credential validation.
- Schema-aware JSON Produce: table-metadata-based record mapping, JSON decoding, validation, and conversion to Fluss rows.
**Delivery**
- End-to-end validation of DDL → Metadata → Kafka producer → Fluss readback, including schema handling, acknowledgements, authentication, and failure cases.
- Configuration and compatibility documentation, supported API/version boundaries, DDL/JSON examples, and Fluss RPC/server regression coverage.
### Future Work
Outside this Basic Produce milestone:
- Idempotent and transactional Produce.
- Bounded Produce admission and flow control.
- Kafka consumer and Fetch compatibility.
- Further authoritative ISR propagation and publisher/cache hardening beyond Metadata discovery.
### Review workflow
1. Create a child issue with a bounded scope, acceptance criteria, and explicit dependencies.
2. Link a focused implementation PR to that child issue; do not close the umbrella from an individual PR.
3. Review Kafka feature issues in dependency order. Keep dependent PRs as drafts and use incremental comparisons while prerequisites are unmerged.
4. As prerequisites merge, rebase dependent branches onto the updated `main` and validate again. Record completion through the child issues, then scope the next unit.
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the scoped child issue #4264, then follow the stated dependency order through #4265, #4275, and #4266 before taking on Basic Produce work. Done means a bounded child issue has reviewed acceptance criteria, a focused implementation PR, and a merged result; this umbrella is not itself a self-contained implementation task.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100