apache / apache/fluss

[kafka] Transcode mapped raw and string records to Arrow

Open
#4,282 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Purpose

Part of #4185. PR06 of the Kafka compatibility sequence.

Use the DDL mapping contract to decode raw/string keys and values, assemble physical rows with optional timestamps and ordered headers, and encode owned native Arrow log bytes.

### Acceptance criteria

- [ ] Raw bytes and strict UTF-8 strings map to the configured physical columns.
- [ ] Nullable values, empty bytes, NOT NULL violations and duplicate/nullable headers are handled correctly.
- [ ] The native batch uses the table schema ID and remains readable after Arrow resources close.
- [ ] Arrow writers and allocators close on encoding failures.

### Dependencies

PR05 Produce protocol (#4281); reuses the PR03 DDL contract (#4277 / #4275).

### Scope boundary

No JSON decoding, gateway append, protocol registration or core module changes.

### Implementation

Draft PR #4285: https://github.com/apache/fluss/pull/4285

Java 11 validation: 75 unit tests and 2 integration tests passed at `503136295d6bbaed81b7e7845dceb2d64db2d9d4`. Review and merge remain pending.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing draft PR #4285, the DDL mapping contract from #4277/#4275, and Produce protocol PR #4281. Check the 75 unit and 2 integration tests against the acceptance criteria, including nullable values, headers, schema IDs, resource cleanup, and readability after Arrow resources close.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.