[FEATURE] Support adding nullable columns to Lance tables via Gravitino API
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Describe the feature
Support `TableChange.AddColumn` for Lance tables in the Generic Lakehouse Catalog through the Gravitino table API and Java client.
Multiple `AddColumn` changes in one request should be applied in a single Lance schema commit. Existing rows should expose the newly added nullable columns as `NULL`.
### Motivation
Schema evolution is a common requirement for Lance tables. Currently, users cannot add Lance columns through Gravitino and must modify the dataset through an external Lance client, which may cause the Lance schema and Gravitino metadata to become inconsistent.
Providing this capability through Gravitino keeps schema changes and table metadata under the same catalog operation.
### Describe the solution
- Validate all changes before modifying the Lance dataset.
- Initially support only top-level, nullable columns appended to the end of the schema.
- Reject default values, auto-increment columns, custom positions, and nested columns.
- Reject requests that combine `AddColumn` with other table-change types.
- Convert all new columns to Arrow fields and call `Dataset.addColumns` once.
- Verify the Lance schema before and after the operation.
- Persist the new columns and Lance dataset version in Gravitino with optimistic-lock protection.
- Attempt to remove the added Lance columns if result validation or Gravitino metadata persistence fails.
- Add unit and integration tests covering batching, conflicts, failure recovery, and `NULL` backfill.
- Document the supported behavior and limitations.
### Additional context
The first phase does not include:
- Nested columns
- Non-nullable columns
- Default-value backfill
- `FIRST` or `AFTER` positions
- Auto-increment columns
- A Lance REST `/add_columns` endpoint
Lance and the Gravitino entity store do not share a distributed transaction, so failure compensation is best-effort.
Contributor guide
Research direction
Start by tracing the Generic Lakehouse Catalog handling of TableChange.AddColumn and the Java client, then inspect the Dataset.addColumns integration. Review how Arrow fields and Lance dataset versions are currently validated and persisted. Done means supported nullable top-level columns are batched into one schema commit, metadata is protected, failures are compensated where possible, and the requested unit, integration, and documentation coverage is present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100