apache / apache/gravitino

[FEATURE] Support adding nullable columns to Lance tables via Gravitino API

Open
#12,382 0 comments 0 reactions 0 assignees View on GitHub
2.0.0 feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the feature

Support `TableChange.AddColumn` for Lance tables in the Generic Lakehouse Catalog through the Gravitino table API and Java client.

Multiple `AddColumn` changes in one request should be applied in a single Lance schema commit. Existing rows should expose the newly added nullable columns as `NULL`.

### Motivation

Schema evolution is a common requirement for Lance tables. Currently, users cannot add Lance columns through Gravitino and must modify the dataset through an external Lance client, which may cause the Lance schema and Gravitino metadata to become inconsistent.

Providing this capability through Gravitino keeps schema changes and table metadata under the same catalog operation.

### Describe the solution

- Validate all changes before modifying the Lance dataset.
- Initially support only top-level, nullable columns appended to the end of the schema.
- Reject default values, auto-increment columns, custom positions, and nested columns.
- Reject requests that combine `AddColumn` with other table-change types.
- Convert all new columns to Arrow fields and call `Dataset.addColumns` once.
- Verify the Lance schema before and after the operation.
- Persist the new columns and Lance dataset version in Gravitino with optimistic-lock protection.
- Attempt to remove the added Lance columns if result validation or Gravitino metadata persistence fails.
- Add unit and integration tests covering batching, conflicts, failure recovery, and `NULL` backfill.
- Document the supported behavior and limitations.

### Additional context

The first phase does not include:

- Nested columns
- Non-nullable columns
- Default-value backfill
- `FIRST` or `AFTER` positions
- Auto-increment columns
- A Lance REST `/add_columns` endpoint

Lance and the Gravitino entity store do not share a distributed transaction, so failure compensation is best-effort.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Generic Lakehouse Catalog handling of TableChange.AddColumn and the Java client, then inspect the Dataset.addColumns integration. Review how Arrow fields and Lance dataset versions are currently validated and persisted. Done means supported nullable top-level columns are batched into one schema commit, metadata is protected, failures are compensated where possible, and the requested unit, integration, and documentation coverage is present.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.