[kafka] Transcode mapped raw and string records to Arrow
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Purpose
Part of #4185. PR06 of the Kafka compatibility sequence.
Use the DDL mapping contract to decode raw/string keys and values, assemble physical rows with optional timestamps and ordered headers, and encode owned native Arrow log bytes.
### Acceptance criteria
- [ ] Raw bytes and strict UTF-8 strings map to the configured physical columns.
- [ ] Nullable values, empty bytes, NOT NULL violations and duplicate/nullable headers are handled correctly.
- [ ] The native batch uses the table schema ID and remains readable after Arrow resources close.
- [ ] Arrow writers and allocators close on encoding failures.
### Dependencies
PR05 Produce protocol (#4281); reuses the PR03 DDL contract (#4277 / #4275).
### Scope boundary
No JSON decoding, gateway append, protocol registration or core module changes.
### Implementation
Draft PR #4285: https://github.com/apache/fluss/pull/4285
Java 11 validation: 75 unit tests and 2 integration tests passed at `503136295d6bbaed81b7e7845dceb2d64db2d9d4`. Review and merge remain pending.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing draft PR #4285, the DDL mapping contract from #4277/#4275, and Produce protocol PR #4281. Check the 75 unit and 2 integration tests against the acceptance criteria, including nullable values, headers, schema IDs, resource cleanup, and readability after Arrow resources close.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100