[Feature](lance) Add namespace and table DDL support
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
## Parent
Tracked by #66340. The baseline Lance integration in #65730 is read-only: it supports namespace/table discovery and table description, but it does not expose Lance metadata mutation operations.
## Current status
Doris currently maps the Lance Namespace SDK's read APIs as follows:
- `SHOW DATABASES` uses `ListNamespaces`.
- `SHOW TABLES` uses `ListTables`.
- `DESC` and table loading use `DescribeTable`.
`LanceExternalCatalog` does not currently provide `ExternalMetadataOps`, so namespace/table creation, schema changes, rename, and drop operations are not supported.
## Scope for Doris 4.2
- [ ] Support `CREATE DATABASE` by creating a Lance Namespace.
- [ ] Support `DROP DATABASE` by dropping a Lance Namespace with well-defined non-empty and `IF EXISTS` behavior.
- [ ] Support `CREATE TABLE` by creating a Lance dataset/table and registering the catalog metadata.
- [ ] Support `SHOW CREATE TABLE` with an accurate Doris representation of the Lance schema and relevant table properties.
- [ ] Support `ALTER TABLE` schema evolution, including add, alter/backfill, rename, and drop column operations supported by Lance.
- [ ] Support `RENAME TABLE`.
- [ ] Support `DROP TABLE`, with explicit semantics for deleting the underlying dataset versus deregistering only the catalog entry.
- [ ] Refresh Doris metadata caches after successful mutations.
- [ ] Define behavior for filesystem and REST Namespace catalogs, authorization, concurrent updates, failures, and version consistency.
Row-level DML (`INSERT`, `UPDATE`, and `DELETE`) is not part of this issue.
## Completion criteria
- [ ] A Lance implementation of Doris external metadata operations is integrated with FE DDL statements.
- [ ] Filesystem and REST Namespace behavior is covered where the corresponding Lance implementation supports the operation.
- [ ] FE unit tests and regression tests cover success, `IF EXISTS`/`IF NOT EXISTS`, conflicts, unsupported operations, and cache refresh.
- [ ] User documentation describes supported syntax, catalog-specific behavior, data-deletion semantics, and limitations.
- [ ] The implementation is merged into the Doris 4.2 branch and verified with representative Lance catalogs.
Contributor guide
Assessment
This issue has not been assessed yet.