[FEATURE] Support named topic data layouts (message schemas) for messaging catalogs
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Describe the feature
Gravitino can catalog Kafka topics today, but the catalog only shows that a topic exists, not what is inside it. The messages on a topic are Protobuf, Avro, or JSON, usually with a schema registered in a schema registry, and none of that appears in the catalog. An engineer who finds a topic in Gravitino still has to dig through a registry, a wiki, or tribal knowledge to learn what the records look like.
This feature implements the reserved `DataLayout` placeholder so topics can carry named message schema metadata, conventionally `key` and `value`. Each layout can describe:
- a schema registry reference: `schemaUri`, `schemaSubject`, `schemaVersion`, `schemaId`
- inline schema text (Avro JSON, `.proto` source, JSON Schema)
- a fully qualified type name (for example a Protobuf message FQN)
- a payload format (`protobuf`, `avro`, `json`) and vendor-specific properties
Layouts are exposed through the Java API (`Topic#dataLayouts()`, `TopicChange.updateDataLayout` / `removeDataLayout` / `removeDataLayouts`), the REST API (`dataLayouts` on topic create/response, three new update operations), and the Java client. They persist in Gravitino's entity store; the Kafka catalog accepts them as entity-store-only metadata since Kafka has no broker-side schema storage.
### Motivation
Topic message schemas are the missing piece of messaging metadata. Without them, Gravitino answers "what topics exist" but not "what data flows through them", which limits its value for discovery, lineage, and governance over streaming data. Example use cases: a discovery tool showing that the `orders` topic carries the Protobuf message `com.example.Order` registered under the `order-value` subject, or a governance process flagging topics with no declared schema.
The layout fields map directly onto schema registry concepts (Confluent Schema Registry, Apicurio, AWS Glue). This issue covers the declarative data model only. The intent is to follow up with registry integration: a `schema.registry.url` property on the Kafka catalog enabling automatic import of key/value layouts from registry subjects, validation of referenced subjects, and eventually drift detection. Landing the model first keeps the registry connectivity design (auth, caching, registry-down behavior) reviewable on its own.
### Design note
Layouts form a named map because messaging payloads carry separate key and value schemas (Schema Registry's `-key` / `-value` subjects). A singular layout cannot express the key schema and would force a breaking migration of a released contract later. Implementing this reshapes `TopicCatalog#createTopic`'s `dataLayout` parameter to `Map`. The existing `DataLayout` is an `@Unstable` empty marker interface documented as "currently not implemented, only reserved as a placeholder", and `TopicCatalog` is `@Evolving`, so this is the intended point in the API lifecycle for such a reshape. An implementation is ready and will be submitted as a PR referencing this issue.
Contributor guide
Research direction
Start with the reserved DataLayout interface and trace TopicCatalog#createTopic, Topic#dataLayouts(), and the TopicChange operations described in the issue. Review the REST API, Java client, and entity-store paths for topic metadata, then verify that named key/value layouts and their schema fields are exposed and persisted across each surface. Done means the declarative model works without requiring schema-registry connectivity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100