apache / apache/doris

[Bug](lance) Vector search misses a later same-column index matching the query metric

Open
#67,117 3 comments 0 reactions 1 assignee Claimed by @FANNG1 View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

## Problem

Doris may fall back to Flat Search even when a metric-compatible Lance vector index exists on the queried column.

Lance permits multiple named logical vector indexes on the same vector column. For example:

```text
vec_l2 column=vector, metric=L2
vec_cosine column=vector, metric=COSINE
```

For a `vector_search()` request with `metric=cosine`, Doris should be able to select `vec_cosine`. Currently, if `vec_l2` appears first in the index metadata returned to the FE, Doris selects all segments belonging to `vec_l2`, checks their metric afterwards, and falls back to fragment-based search when the metric does not match. It does not continue looking for `vec_cosine`.

The query result remains correct because Lance executes Flat Search with the requested metric, but the available ANN index is silently missed. Query latency and resource usage can therefore depend on index metadata ordering.

## Current implementation

`LanceScanNode.selectIndexSegments()` first locks onto the first logical index covering the vector field:

```java
if (selectedIndexName == null) {
selectedIndexName = segment.getIndexName();
}
if (selectedIndexName.equals(segment.getIndexName())) {
selectedSegments.add(segment);
}
```

Only after that selection does `createIndexSegmentSplits()` call `metricMatches()`:

```java
List matchingSegments = selectIndexSegments(
metadata.getIndexSegments(), vectorFieldId);
if (matchingSegments.isEmpty() || !metricMatches(vectorSearchParam, matchingSegments)) {
return Optional.empty();
}
```

Relevant code:

- `fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java`
- `fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceMetadataLoader.java`

The FE already loads the information needed to make a metric-aware decision:

- logical index name
- indexed field IDs
- physical segment UUIDs
- index metric
- fragment coverage

This is not a Lance index-format limitation. Doris can select a logical index by sending that index's physical segment UUIDs to the BE; no new Lance `index_name` scanner parameter is required for automatic metric-aware selection.

## Expected behavior

When `use_index=true`, Doris should:

1. Group physical index segments by logical index name.
2. Keep logical indexes covering the requested vector field.
3. Resolve the requested metric using Doris semantics (`L2` when the query metric is omitted or `DEFAULT`).
4. Select a logical index whose segments use the requested metric.
5. Route only that logical index's segment UUIDs to the BE.
6. Fall back to Flat Search only when no compatible logical index can be planned safely.

If more than one logical index matches both the field and metric, Doris should either define a deterministic documented policy or later expose an optional `index_name` query property. Explicit user selection is a separate enhancement; it is not required to fix the missed compatible-index case.

Legacy index metadata without a metric or fragment bitmap should continue to use the existing conservative fallback unless a safe compatibility path is defined.

## Reproduction

1. Create a Lance dataset with one vector column.
2. Create two named logical indexes on that column, one using L2 and one using cosine, ensuring the L2 index is returned first by `describeIndices()`.
3. Run Doris `vector_search()` with `metric=cosine` and `use_index=true`.
4. Inspect `EXPLAIN`.

Current behavior:

```text
lanceSearchIndexSegments=0
```

and the query falls back to fragment search, even though the cosine index exists.

Expected behavior:

```text
lanceSearchIndexSegments>0
```

with only the `vec_cosine` physical segments assigned to indexed splits.

## Acceptance criteria

- [ ] FE unit test: L2 index appears first, cosine index second, and a cosine query selects the cosine segments.
- [ ] FE unit test: reversing metadata order does not change the selected metric-compatible logical index.
- [ ] FE unit test: segments from different logical index names are never mixed in one index plan.
- [ ] FE unit test: no compatible metric still produces the existing Flat Search fallback.
- [ ] FE unit test: omitted/`DEFAULT` metric follows the documented Doris L2 default.
- [ ] Regression test proves physical indexed execution, using both `EXPLAIN` segment planning and an `nprobes`/`ef` discriminator rather than only comparing indexed and flat results.
- [ ] Documentation no longer recommends one index per vector column solely because Doris cannot find a later metric-compatible index.

## Related

- #66340
- apache/doris#67039 currently uses one vector column per matrix cell to avoid this selection limitation; it expands coverage but does not change the production selection logic.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.