apache / apache/gravitino

[Improvement] Add lightweight table metadata probing for Lance REST describeTable

Open
#11,442 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

Gravitino should provide a lightweight table metadata probing interface for use cases that do not need full table schema loading.

Currently, Gravitino REST and Lance REST share the same `TableCatalog.loadTable` path. This creates a coupling problem:

- Gravitino REST wants to show the latest table schema, so `loadTable` may reasonably trigger schema loading or schema repair from the underlying storage.
- Lance REST `describeTable` is often used as a lightweight probe. In most cases it only needs location, table properties, and storage options, especially when `DescribeTableRequest.load_detailed_metadata=false`.

Because both paths use the same `loadTable` method, improving Gravitino REST schema correctness can unintentionally make Lance REST lightweight describe requests open the underlying Lance dataset and refresh/write Gravitino table columns.

This is the main issue: a full schema-loading API and a lightweight metadata-probing API are currently the same operation.

### How should we improve?

Add a lightweight table metadata probing interface in Gravitino.

This interface should let callers fetch basic table metadata, such as location/properties/storage options, without triggering schema loading, underlying storage access, or lazy schema repair.

Lance REST `describeTable(load_detailed_metadata=false)` can use this lightweight probing path, while Gravitino REST and detailed Lance REST requests can continue using the full table loading path when schema information is required.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Gravitino REST and Lance REST through TableCatalog.loadTable and the Lance REST describeTable entry point, including the load_detailed_metadata=false path. Define where a lightweight metadata probe should separate from full loading; done means basic metadata is returned without schema loading or storage access, while detailed requests still load schema information.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.