apache / apache/fluss

[kafka] Define DDL table mapping for Kafka compatibility

Open
#4,275 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Parent issue

Part of #4185 — Kafka Protocols Compatibility: Basic Produce.

### Motivation

Define the DDL table mapping contract before Metadata and Produce so both APIs agree on which Fluss tables can be exposed as Kafka topics and how records map to physical columns. This is PR03 in the foundation sequence; Metadata is PR04 (#4266).

### Scope

- Map a topic name to the same-named table in the configured Kafka database, table IDs to stable topic IDs, and Kafka partitions to Fluss buckets.
- Define the initial supported table shape: non-partitioned Arrow log tables without a primary key.
- Resolve `kafka.*` custom properties for raw/string key and value formats, field projections, and optional timestamp/headers columns.
- Validate field names, types, projection conflicts, and unsupported mapping options through a shared resolver that Metadata and later Produce can reuse.
- Cover valid/invalid mappings and a native Fluss create-table → metadata-readback round trip.

### Acceptance criteria

- [ ] The mapping is defined by the existing table schema and custom properties, without a mandatory four-column envelope.
- [ ] Valid raw/string mappings resolve deterministically; invalid table kinds, fields, types and overlapping projections are rejected.
- [ ] Topic identity remains stable during a table lifetime and changes when the table is recreated.
- [ ] Mapping properties survive native Fluss table creation and metadata retrieval.
- [ ] Relevant unit/integration tests, style and license checks pass, and the focused implementation is reviewed and merged.

### Dependencies

Depends on #4265, which builds on #4264. This issue is a prerequisite for Metadata (#4266) and subsequent Produce capabilities.

### Out of scope

Kafka CreateTopics/DeleteTopics, automatic table creation, Metadata request handling, Produce, byte decoding, Arrow encoding, JSON conversion, and authentication.

### Implementation and review

Implementation PR: #4277 (Draft), from `gyang94:codex/kafka-ddl-table-mapping`, based directly on PR02 (#4260). Local Java 11 validation passed 47 unit tests and 2 integration tests, including the native DDL round trip, plus Checkstyle, Spotless and RAT checks. These results do not imply remote CI or review completion.

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the implementation PR #4277 and the prerequisite issues #4265 and #4264 first. Check the shared mapping resolver and its valid/invalid mapping tests, then run the native Fluss create-table to metadata-readback round trip. Done means the acceptance criteria pass and the focused implementation is reviewed and merged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.