apache / apache/doris

[Feature](lance) Add namespace and table DDL support

Open
#66,516 1 comment 0 reactions 1 assignee Claimed by @wangzhaobo957-cloud View on GitHub
area/catalog kind/feature
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

## Parent

Tracked by #66340. The baseline Lance integration in #65730 is read-only: it supports namespace/table discovery and table description, but it does not expose Lance metadata mutation operations.

## Current status

Doris currently maps the Lance Namespace SDK's read APIs as follows:

- `SHOW DATABASES` uses `ListNamespaces`.
- `SHOW TABLES` uses `ListTables`.
- `DESC` and table loading use `DescribeTable`.

`LanceExternalCatalog` does not currently provide `ExternalMetadataOps`, so namespace/table creation, schema changes, rename, and drop operations are not supported.

## Scope for Doris 4.2

- [ ] Support `CREATE DATABASE` by creating a Lance Namespace.
- [ ] Support `DROP DATABASE` by dropping a Lance Namespace with well-defined non-empty and `IF EXISTS` behavior.
- [ ] Support `CREATE TABLE` by creating a Lance dataset/table and registering the catalog metadata.
- [ ] Support `SHOW CREATE TABLE` with an accurate Doris representation of the Lance schema and relevant table properties.
- [ ] Support `ALTER TABLE` schema evolution, including add, alter/backfill, rename, and drop column operations supported by Lance.
- [ ] Support `RENAME TABLE`.
- [ ] Support `DROP TABLE`, with explicit semantics for deleting the underlying dataset versus deregistering only the catalog entry.
- [ ] Refresh Doris metadata caches after successful mutations.
- [ ] Define behavior for filesystem and REST Namespace catalogs, authorization, concurrent updates, failures, and version consistency.

Row-level DML (`INSERT`, `UPDATE`, and `DELETE`) is not part of this issue.

## Completion criteria

- [ ] A Lance implementation of Doris external metadata operations is integrated with FE DDL statements.
- [ ] Filesystem and REST Namespace behavior is covered where the corresponding Lance implementation supports the operation.
- [ ] FE unit tests and regression tests cover success, `IF EXISTS`/`IF NOT EXISTS`, conflicts, unsupported operations, and cache refresh.
- [ ] User documentation describes supported syntax, catalog-specific behavior, data-deletion semantics, and limitations.
- [ ] The implementation is merged into the Doris 4.2 branch and verified with representative Lance catalogs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.