[Tracking] Lance feature matrix for Doris 4.2

## Goal

Plan and track the Doris 4.2 support scope for Lance Catalog, Lance dataset access, and vector search.

## Doris 4.2 work items

Check an item only after implementation, tests, documentation, and verification are complete.

### Catalog integration

- [x] **Filesystem Lance Catalog with a local path**
- [x] **Filesystem Lance Catalog with file://**
- [x] **Filesystem Lance Catalog with s3://**
- [x] **Lance REST Namespace Catalog**

### Authentication and credentials

- [x] **Bearer Token authentication**
- [x] **API Key authentication**
- [x] **Custom HTTP headers**
- [x] **Temporary storage credentials returned by the Namespace service**

### Namespace and metadata

- [x] **Map multilevel Lance Namespaces to Doris databases**
- [x] **SHOW DATABASES**
- [x] **SHOW TABLES**
- [x] **DESC**

### Namespace and table DDL

The baseline implementation in #65730 is read-only. `SHOW DATABASES`, `SHOW TABLES`, and `DESC` are supported through `ListNamespaces`, `ListTables`, and `DescribeTable`, while `LanceExternalCatalog` does not currently provide `ExternalMetadataOps` or invoke namespace/table mutation APIs. The remaining DDL work below is planned for Doris 4.2 and tracked by #66516.

- [ ] **CREATE DATABASE / Lance Namespace**
- [ ] **DROP DATABASE / Lance Namespace**
- [ ] **CREATE TABLE / Lance dataset and catalog registration**
- [ ] **SHOW CREATE TABLE**
- [ ] **ALTER TABLE schema evolution** — Add, alter/backfill, rename, and drop columns where supported by Lance.
- [ ] **RENAME TABLE**
- [ ] **DROP TABLE / deregister table** — Define whether each operation deletes the underlying dataset or only removes the catalog entry.
- [ ] **Metadata cache refresh and failure semantics for DDL**

### Versioning and consistency

- [x] **Pin the dataset version during planning** — Preserve snapshot consistency for the full query lifecycle.
- [ ] **REST Namespace managed versioning in the BE reader**

### Scan execution

- [x] **Lance reader in File Scanner V2**
- [x] **Parallel scan by Lance Fragment**
- [x] **Column pruning**
- [x] **Retain unsupported predicates for execution in Doris** — Keep predicates in Doris whenever pushdown cannot preserve SQL semantics.

### Predicate pushdown

- [x] **Push semantically compatible scalar predicates to Lance** — Push only predicates whose semantics are compatible with Lance.
- [x] **SQL filter for vector_search()**
- [x] **Substrait filter for regular Lance scans**

### Observability

- [x] **Show pushed predicates in EXPLAIN** — Expose pushdown decisions for troubleshooting and verification.

### Direct dataset access

- [x] **Read a Lance dataset through the s3() TVF**
- [x] **Read a Lance dataset through the local() TVF**

### Paths and splits

- [x] **Split one Lance dataset into multiple Fragment splits** — Generate parallel scan splits from Lance Fragments.
- [ ] **Expand glob patterns to multiple Lance datasets** — Resolve one path pattern into multiple datasets.
- [ ] **Read multiple Lance datasets in one TVF invocation** — Plan and scan more than one dataset in a single TVF call.
- [ ] **Support additional file TVF entry points and broader path handling**

### Read operations

- [x] **Read Lance datasets**

### Search modes

- [x] **ANN index search**
- [x] **Flat exact search**
- [ ] **Radius/Range Search** — Expose lower and upper distance bounds.
- [ ] **Full-text search with BM25** — Expose standalone BM25 full-text search.
- [ ] **Multi-vector Search** — Support multi-vector columns and multiple sub-vectors in one logical query.

### Vector index algorithms

- [ ] **IVF_FLAT query support**
- [ ] **IVF_SQ query support**
- [x] **IVF_PQ query support**
- [ ] **IVF_HNSW_FLAT query support**
- [ ] **IVF_HNSW_SQ query support**
- [ ] **IVF_HNSW_PQ query support**

### Distance metrics

- [x] **L2**
- [x] **Cosine**
- [x] **Dot Product**
- [x] **Hamming** — Support compatible packed binary vectors and indexes.

### Data types

- [x] **FixedSizeList**
- [x] **FixedSizeList**
- [x] **FixedSizeList** — Define compatible ANN index and distance-metric combinations.
- [x] **FixedSizeList** — Support UInt8 and compatible packed binary-vector use cases.
- [x] **FixedSizeList**
- [ ] **List> for multi-vector columns** — Map nested vector columns and define multi-vector scoring semantics.
- [ ] **Blob v2**
- [ ] **Arrow JSON Extension**
- [ ] **BFloat16 Extension**
- [ ] **Arrow null**
- [ ] **Arrow duration**

### Query input

- [x] **One query vector per search**
- [ ] **Multiple sub-vectors in one logical query**

### Filtering

- [x] **Scalar pre-filter** — Apply scalar predicates before vector candidate selection.
- [x] **Scalar post-filter** — Allow scalar predicates to run after vector candidate selection.

### Index lifecycle

- [ ] **Create vector indexes**
- [ ] **Create scalar indexes**
- [ ] **Show the number of indexes**
- [ ] **Show index names, UUIDs, columns, types, and dataset versions**
- [ ] **Replace or rebuild an index with the same name**
- [ ] **Drop indexes by name**

### Index selection

- [x] **Automatically select an index for the target vector column** — Use Lance index selection for the requested vector column.

### ANN tuning

- [x] **nprobes**
- [x] **refine_factor**
- [x] **ef**
- [x] **Disable the ANN index with use_index=false**

### Result output

- [x] **Top-K**
- [x] **Offset**
- [x] **Return distance through the _distance virtual column** — Document exact, approximate, and refined distance semantics.

### Tests and documentation

- [x] **FE unit tests**
- [x] **BE unit tests**
- [x] **Regression tests for Catalog queries, predicate pushdown, file TVFs, and vector search**
- [x] **Lance Catalog user documentation** — Publish the supported scope, configuration, examples, and limitations.

## Explicitly not planned for Doris 4.2

These capabilities are excluded from the Doris 4.2 plan.

| Category | Capability | Reason |
|---|---|---|
| Read and write operations | 🚫 **INSERT into Lance** | Doris 4.2 does not include a Lance write path for appending rows. |
| Read and write operations | 🚫 **UPDATE Lance data** | Doris 4.2 does not include a Lance write path for modifying and committing rows. |
| Read and write operations | 🚫 **DELETE Lance data** | Doris 4.2 does not include a Lance write path for deleting and committing rows. |
| Search modes | 🚫 **Hybrid Search (Vector + BM25)** | LanceDB can orchestrate two searches at a higher layer, but Lance Scanner/lance-c makes nearest and FTS mutually exclusive. |
| Search modes | 🚫 **Batch Search** | The current Scanner/lance-c nearest API accepts only one independent query vector. |
| Search modes | 🚫 **Joint search and scoring across multiple vector columns** | One Lance vector query accepts only one vector column. |
| Distance metrics | 🚫 **Custom distance functions** | Lance ANN accepts only its built-in distance enum and cannot accept a callback, SQL UDF, or arbitrary expression. |
| Data types | 🚫 **Vector element types outside the current nearest C ABI** | The current nearest C ABI defines Float16, Float32, Float64, UInt8, and Int8 only. |
| Index selection | 🚫 **Select index_name at query time** | Lance Scanner Query and lance-c nearest do not expose an index-name parameter. |
| Index selection | 🚫 **Select index_type at query time** | The current query path does not expose an index-type selector. |

## Completion criteria

- The capability is merged into the Doris 4.2 branch.
- FE/BE unit tests and regression tests cover the capability.
- User documentation describes the supported behavior, parameter semantics, and limitations.
- The capability is verified with representative Lance datasets and object-storage access where applicable.

Review the baseline implementation in #65730 and the remaining DDL work tracked by #66516, then choose a specific unchecked capability rather than treating this tracking issue as one task. Start with the relevant FE/BE unit and regression tests mentioned in the issue. Done means the capability is merged into Doris 4.2, tested, documented, and verified with representative Lance datasets where applicable.

Written by the indexing model from the issue text.