matrixorigin / matrixorigin/matrixone

[Bug]: IVFFLAT cache keeps the first L2 function and misfilters later l2/l2sq ranges

Open
#29,038 4 comments 0 reactions 0 assignees View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I have checked the existing issues.

### Branch Name

main

### Commit ID

01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90

### Other Environment Information

- Local multi-CN launch (`etc/launch-multi-cn/launch.toml`), SQL ports 16001 and 16002
- Synchronous IVFFLAT, `lists=1`, `vector_l2_ops`
- 120 rows in three exact distance shells; no approximate-list recall variable

### Actual Behavior

An IVFFLAT L2 index remembers whichever equivalent SQL distance function uses the cached index first. A later query using the other function evaluates pushed distance ranges in the first function's unit and can return an empty result.

Both directions reproduce:

1. `l2_distance` runs first and correctly returns ten rows at distance `200`; the next `l2_distance_sq` query for `22500 < distance_sq < 62500` returns zero rows instead of the same ten rows at squared distance `40000`.
2. `l2_distance_sq` runs first and correctly returns ten rows at squared distance `40000`; the next `l2_distance` query for `150 < distance < 250` returns zero rows instead of the same ten rows at distance `200`.

The alternate-function query returns zero in `mode=post`, `mode=include`, and `mode=pre`. `mode=force` returns the expected ten rows, so the source data and SQL predicates are valid.

This was repeated with a previously unused table/index cache key for every attempt: two CN endpoints × both first-function directions × three independent indexes × three IVF modes = **36/36 incorrect alternate-function results**. Every first-function execution and every exact control was correct.

### Expected Behavior

`l2_distance` and `l2_distance_sq` are both supported by `vector_l2_ops` and `vector_l2sq_ops`. Each query must use its own function's unit when applying `DistRange`, regardless of which function first loaded or explained the cached IVFFLAT index.

### Steps to Reproduce

```sql
set experimental_ivf_index = 1;
drop database if exists ivf_l2_func_cache_repro;
create database ivf_l2_func_cache_repro;
use ivf_l2_func_cache_repro;

create table t(id int primary key, grp int, v vecf32(2));
insert into t
select result,
floor((result - 1) / 40),
cast(concat('[', floor((result - 1) / 40) * 200, ',0]') as vecf32(2))
from generate_series(1, 120) g;

create index idx using ivfflat on t(v)
lists = 1 op_type 'vector_l2_ops';

-- First use of this index: correct, ten rows at distance 200.
select id, grp, l2_distance(v, '[0,0]') d
from t
where l2_distance(v, '[0,0]') > 150
and l2_distance(v, '[0,0]') < 250
order by l2_distance(v, '[0,0]')
limit 10 by rank with option 'mode=post';

-- Same vectors and equivalent squared interval: incorrectly returns zero.
select id, grp, l2_distance_sq(v, '[0,0]') d
from t
where l2_distance_sq(v, '[0,0]') > 22500
and l2_distance_sq(v, '[0,0]') < 62500
order by l2_distance_sq(v, '[0,0]')
limit 10 by rank with option 'mode=post';

-- Exact control: correctly returns ten rows at squared distance 40000.
select id, grp, l2_distance_sq(v, '[0,0]') d
from t
where l2_distance_sq(v, '[0,0]') > 22500
and l2_distance_sq(v, '[0,0]') < 62500
order by l2_distance_sq(v, '[0,0]')
limit 10 by rank with option 'mode=force';
```

To reproduce the reverse direction, recreate the table/index with a new name and execute the squared-distance query first.

### Additional information

The plan reader constructs a per-query `IndexTableConfig` with `OrigFuncName = r.spec.DistanceFunction`, but the IVFFLAT cache key is the centroid hidden table plus index version. The cached search object retains the first `Tblcfg`. In the relation-search path, range filtering calls `filterEntryDistanceRange(..., tblcfg.OrigFuncName, ...)`, while returned scores use the current request's `RuntimeConfig.OrigFuncName`. This split explains why ordinary scores remain correctly labeled but the pushed range is evaluated using stale L2 versus squared-L2 conversion.

This is the corrected independent defect found while expanding #28943. The earlier #29037 was closed after fresh-name controls disproved its original static-range diagnosis.

Contributor guide

Open the contributing guide

Research direction

Start with the plan reader's per-query IndexTableConfig and trace the relation-search path through filterEntryDistanceRange, comparing OrigFuncName with RuntimeConfig. Use the SQL reproduction in the issue with both l2_distance and l2_distance_sq query orders, then verify that each alternate-function query returns the same ten rows in post, include, and pre modes while force remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.