apache / apache/gravitino

[FEATURE] Support TableSchema Catalog to manage table schema (schema registry)

Open
#5,230 4 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the feature

In Kafka and Fileset, we may need a table schema to deserialize data. We can manage the external schema registry in Gravitino.

### Motivation

- Manage the table schema in Gravitino.
- Using SQL to read the Kafka topic or Fileset when binding the table schema to the topic or fileset.

### Describe the solution

We can introduce a TableSchemaCatalog to manage the TableSchema.

We can bind a table schema such as `catalog.schema.table-schema` to a topic or fileset when needed. So we can get the table schema from the external schema registry. We can also add a schema registry managed by gravitino, so we can directory save the table schema to the gravitino metastore.

![img_v3_02fu_4b0b7b27-c424-4a5b-a0fc-42c6fb54318l](https://github.com/user-attachments/assets/31950e21-464d-4795-bfef-18a52eb6b188)

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the proposed TableSchemaCatalog and the binding flow described for Kafka topics and Filesets. Review the linked design image and clarify the catalog, external schema registry, SQL access, and metastore storage requirements; done means an agreed design and implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, sql
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.