apache / apache/fluss

[kafka] Implement Metadata API for pre-created Fluss tables

Open
#4,266 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Parent issue

Part of #4185 — Kafka Protocols Compatibility: Basic Produce.

### Motivation

Standard Kafka producers need metadata discovery before they can send records. Expose tables created through Fluss DDL as Kafka topics while preserving the DDL-defined table and schema lifecycle.

### Scope

- Implement Kafka Metadata v0-v11 over a narrow metadata backend and the TabletServer gateway.
- Reuse the table/topic mapping and schema validation contract from #4275; return brokers, partitions, leaders, replica assignments, ISR, and offline replicas for compatible tables.
- Preserve the Kafka listener context when resolving advertised endpoints.
- Handle invalid or missing topics, unavailable leaders, gateway failures, and table-deletion races.
- Respect authoritative ISR when bucket metadata provides it, with a conservative available-leader fallback for legacy metadata.
- Register Metadata so ApiVersions advertises the implemented capability.

### Acceptance criteria

- [ ] Named-topic, all-topic, and empty-topic discovery follow the DDL mapping contract and return the expected broker and partition metadata.
- [ ] Unsupported table mappings do not affect valid topics in the same request.
- [ ] Version boundaries and invalid requests, including missing names in v10/v11, are handled correctly.
- [ ] Replica/ISR reporting, legacy fallback, and error paths have regression coverage.
- [ ] Relevant handler, mapping, transport, and integration tests and style checks pass.
- [ ] The focused implementation is reviewed and merged after its prerequisites.

### Dependencies

PR04 in the foundation sequence. Depends on the PR03 DDL table mapping contract (#4275), which builds on #4265 and #4264.

### Out of scope

Kafka CreateTopics/DeleteTopics, automatic table creation, Metadata v12+ or ID-only topic lookup, Produce, authentication, and JSON-to-row conversion. Schema-aware JSON Produce is a later issue under #4185; this stage only provides discovery for pre-created tables.

### Implementation and review

Implementation PR: #4261.

The PR remains a draft while its prerequisites are pending. Its description includes an incremental comparison for reviewing Metadata alone.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading implementation PR #4261 and its incremental comparison, then review the PR03 DDL table mapping contract in #4275 and the prerequisites #4265 and #4264. Trace the handler, mapping, transport, and integration tests mentioned in the issue; done means the Metadata v0-v11 acceptance criteria and relevant regression and style checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
api, backend, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.