[Improvement] Lance table creation should inherit catalog/schema-level storage properties (AK/SK, endpoint)
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 315
Description
### What would you like to be improved?
When creating a Lance table, object storage credentials (_lance.storage.*_ properties such as AK/SK, endpoint, region) must be explicitly provided at the table level every time. There is no inheritance mechanism for these properties from the
catalog or schema level.
Currently, only the _location_ property has a fallback chain: table properties → schema _location_ → catalog _location_ (implemented in `GenericCatalogOperations.calculateTableLocation()`). However, _lance.storage.*_ properties are defined only at the
table level via `LanceTableDelegator.tablePropertyEntries()` and `LancePropertiesUtils.getLanceStorageOptions()` only extracts them from table properties.
**Key observations:**
1. `GenericCatalogPropertiesMetadata` only defines `Catalog.PROPERTY_LOCATION `— no _lance.storage.*_ entries exist at the catalog level.
2. `GenericSchemaPropertiesMetadata` only defines `Schema.PROPERTY_LOCATION` — no storage credential entries at the schema level either.
3. `GenericCatalogOperations.createTable() `merges _location_ and _table-format_ into table properties, but never merges any storage credentials from catalog or schema.
4. `LanceTableOperations.createTableInternal()` calls `LancePropertiesUtils.getLanceStorageOptions`(properties), which only filters the _lance.storage._ prefix from the passed-in table properties — if no storage credentials were merged upstream,
they are simply absent.
This means users must redundantly specify the same AK/SK and endpoint on every createTable call, even though these values are typically shared across all tables under the same catalog or schema.
### How should we improve?
Introduce a storage property inheritance chain similar to the existing location fallback:
1. **Define** _lance.storage.*_ **as a prefix property at catalog and schema levels** — add `stringOptionalPropertyPrefixEntry(LANCE_STORAGE_OPTIONS_PREFIX, ...)` to `GenericCatalogPropertiesMetadata` and `GenericSchemaPropertiesMetadata`.
2. **Implement a** `calculateStorageOptions()` **method in** `GenericCatalogOperations` (or similar location) that merges storage properties with a fallback chain: table properties → schema _lance.storage.*_ → catalog _lance.storage.*_. Table-level
properties should take precedence (override) over schema and catalog level.
3. **Merge the resolved storage options into table properties before delegating to** `LanceTableOperations.createTable()`, so that `LancePropertiesUtils.getLanceStorageOptions()` can pick them up as it does today without changes.
4. **Consider security implications** — catalog-level storage credentials are stored in Gravitino metadata. Evaluate whether these should be marked as hidden to avoid exposing them in API responses, similar to how other catalogs handle sensitive
properties.
Contributor guide
Research direction
Start with GenericCatalogPropertiesMetadata, GenericSchemaPropertiesMetadata, and GenericCatalogOperations.createTable(), then compare calculateTableLocation() with LancePropertiesUtils.getLanceStorageOptions() and LanceTableOperations.createTableInternal(). Add the catalog-to-schema-to-table storage-property fallback, preserve table overrides, and verify the metadata exposure and security implications described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100