apache / apache/gravitino

[FEATURE] Support named topic data layouts (message schemas) for messaging catalogs

Open
#11,996 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the feature

Gravitino can catalog Kafka topics today, but the catalog only shows that a topic exists, not what is inside it. The messages on a topic are Protobuf, Avro, or JSON, usually with a schema registered in a schema registry, and none of that appears in the catalog. An engineer who finds a topic in Gravitino still has to dig through a registry, a wiki, or tribal knowledge to learn what the records look like.

This feature implements the reserved `DataLayout` placeholder so topics can carry named message schema metadata, conventionally `key` and `value`. Each layout can describe:

- a schema registry reference: `schemaUri`, `schemaSubject`, `schemaVersion`, `schemaId`
- inline schema text (Avro JSON, `.proto` source, JSON Schema)
- a fully qualified type name (for example a Protobuf message FQN)
- a payload format (`protobuf`, `avro`, `json`) and vendor-specific properties

Layouts are exposed through the Java API (`Topic#dataLayouts()`, `TopicChange.updateDataLayout` / `removeDataLayout` / `removeDataLayouts`), the REST API (`dataLayouts` on topic create/response, three new update operations), and the Java client. They persist in Gravitino's entity store; the Kafka catalog accepts them as entity-store-only metadata since Kafka has no broker-side schema storage.

### Motivation

Topic message schemas are the missing piece of messaging metadata. Without them, Gravitino answers "what topics exist" but not "what data flows through them", which limits its value for discovery, lineage, and governance over streaming data. Example use cases: a discovery tool showing that the `orders` topic carries the Protobuf message `com.example.Order` registered under the `order-value` subject, or a governance process flagging topics with no declared schema.

The layout fields map directly onto schema registry concepts (Confluent Schema Registry, Apicurio, AWS Glue). This issue covers the declarative data model only. The intent is to follow up with registry integration: a `schema.registry.url` property on the Kafka catalog enabling automatic import of key/value layouts from registry subjects, validation of referenced subjects, and eventually drift detection. Landing the model first keeps the registry connectivity design (auth, caching, registry-down behavior) reviewable on its own.

### Design note

Layouts form a named map because messaging payloads carry separate key and value schemas (Schema Registry's `-key` / `-value` subjects). A singular layout cannot express the key schema and would force a breaking migration of a released contract later. Implementing this reshapes `TopicCatalog#createTopic`'s `dataLayout` parameter to `Map`. The existing `DataLayout` is an `@Unstable` empty marker interface documented as "currently not implemented, only reserved as a placeholder", and `TopicCatalog` is `@Evolving`, so this is the intended point in the API lifecycle for such a reshape. An implementation is ready and will be submitted as a PR referencing this issue.

Contributor guide

Open the contributing guide

Research direction

Start with the reserved DataLayout interface and trace TopicCatalog#createTopic, Topic#dataLayouts(), and the TopicChange operations described in the issue. Review the REST API, Java client, and entity-store paths for topic metadata, then verify that named key/value layouts and their schema fields are exposed and persisted across each surface. Done means the declarative model works without requiring schema-registry connectivity.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.