matrixorigin / matrixorigin/matrixone
[Bug]: IVFFLAT cache keeps the first L2 function and misfilters later l2/l2sq ranges
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Is there an existing issue for the same bug?
- [x] I have checked the existing issues.
### Branch Name
main
### Commit ID
01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90
### Other Environment Information
- Local multi-CN launch (`etc/launch-multi-cn/launch.toml`), SQL ports 16001 and 16002
- Synchronous IVFFLAT, `lists=1`, `vector_l2_ops`
- 120 rows in three exact distance shells; no approximate-list recall variable
### Actual Behavior
An IVFFLAT L2 index remembers whichever equivalent SQL distance function uses the cached index first. A later query using the other function evaluates pushed distance ranges in the first function's unit and can return an empty result.
Both directions reproduce:
1. `l2_distance` runs first and correctly returns ten rows at distance `200`; the next `l2_distance_sq` query for `22500 < distance_sq < 62500` returns zero rows instead of the same ten rows at squared distance `40000`.
2. `l2_distance_sq` runs first and correctly returns ten rows at squared distance `40000`; the next `l2_distance` query for `150 < distance < 250` returns zero rows instead of the same ten rows at distance `200`.
The alternate-function query returns zero in `mode=post`, `mode=include`, and `mode=pre`. `mode=force` returns the expected ten rows, so the source data and SQL predicates are valid.
This was repeated with a previously unused table/index cache key for every attempt: two CN endpoints × both first-function directions × three independent indexes × three IVF modes = **36/36 incorrect alternate-function results**. Every first-function execution and every exact control was correct.
### Expected Behavior
`l2_distance` and `l2_distance_sq` are both supported by `vector_l2_ops` and `vector_l2sq_ops`. Each query must use its own function's unit when applying `DistRange`, regardless of which function first loaded or explained the cached IVFFLAT index.
### Steps to Reproduce
```sql
set experimental_ivf_index = 1;
drop database if exists ivf_l2_func_cache_repro;
create database ivf_l2_func_cache_repro;
use ivf_l2_func_cache_repro;
create table t(id int primary key, grp int, v vecf32(2));
insert into t
select result,
floor((result - 1) / 40),
cast(concat('[', floor((result - 1) / 40) * 200, ',0]') as vecf32(2))
from generate_series(1, 120) g;
create index idx using ivfflat on t(v)
lists = 1 op_type 'vector_l2_ops';
-- First use of this index: correct, ten rows at distance 200.
select id, grp, l2_distance(v, '[0,0]') d
from t
where l2_distance(v, '[0,0]') > 150
and l2_distance(v, '[0,0]') < 250
order by l2_distance(v, '[0,0]')
limit 10 by rank with option 'mode=post';
-- Same vectors and equivalent squared interval: incorrectly returns zero.
select id, grp, l2_distance_sq(v, '[0,0]') d
from t
where l2_distance_sq(v, '[0,0]') > 22500
and l2_distance_sq(v, '[0,0]') < 62500
order by l2_distance_sq(v, '[0,0]')
limit 10 by rank with option 'mode=post';
-- Exact control: correctly returns ten rows at squared distance 40000.
select id, grp, l2_distance_sq(v, '[0,0]') d
from t
where l2_distance_sq(v, '[0,0]') > 22500
and l2_distance_sq(v, '[0,0]') < 62500
order by l2_distance_sq(v, '[0,0]')
limit 10 by rank with option 'mode=force';
```
To reproduce the reverse direction, recreate the table/index with a new name and execute the squared-distance query first.
### Additional information
The plan reader constructs a per-query `IndexTableConfig` with `OrigFuncName = r.spec.DistanceFunction`, but the IVFFLAT cache key is the centroid hidden table plus index version. The cached search object retains the first `Tblcfg`. In the relation-search path, range filtering calls `filterEntryDistanceRange(..., tblcfg.OrigFuncName, ...)`, while returned scores use the current request's `RuntimeConfig.OrigFuncName`. This split explains why ordinary scores remain correctly labeled but the pushed range is evaluated using stale L2 versus squared-L2 conversion.
This is the corrected independent defect found while expanding #28943. The earlier #29037 was closed after fresh-name controls disproved its original static-range diagnosis.
Contributor guide
Research direction
Start with the plan reader's per-query IndexTableConfig and trace the relation-search path through filterEntryDistanceRange, comparing OrigFuncName with RuntimeConfig. Use the SQL reproduction in the issue with both l2_distance and l2_distance_sq query orders, then verify that each alternate-function query returns the same ten rows in post, include, and pre modes while force remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100