matrixorigin / matrixorigin/matrixone
[Bug]: LIMIT BY RANK silently ignores query-level nprobe and fudge_factor options
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### MatrixOne version
`01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90` (current `main` under validation)
### Environment
Local distributed deployment with 2 CN, 1 TN, and 1 log service.
### Problem
`LIMIT ... BY RANK WITH OPTION` accepts the documented/proposed query-level `nprobe` and `fudge_factor` options, but the planner silently discards both. Only `mode` is copied into `plan.RankOption`.
Consequently, users cannot tune IVF probe count per query even though the SQL is accepted. The actual NProbe always comes from the session variable `probe_limit`.
### Reproduction
```sql
set experimental_ivf_index = 1;
create database rank_option_repro;
use rank_option_repro;
create table t(id bigint primary key, v vecf32(2));
insert into t select result, concat('[', result, ',', result, ']')
from generate_series(1, 1000) g;
create index ix using ivfflat on t(v)
op_type 'vector_l2_ops' lists=20;
set probe_limit = 5;
explain verbose
select id from t
order by l2_distance(v, '[0,0]')
limit 5 by rank with option 'mode=post', 'nprobe=1';
explain verbose
select id from t
order by l2_distance(v, '[0,0]')
limit 5 by rank with option 'mode=post', 'nprobe=99';
explain verbose
select id from t
order by l2_distance(v, '[0,0]')
limit 5 by rank with option 'mode=post', 'fudge_factor=100';
```
All three plans report:
```text
Vector Index: ix, Metric: l2_distance, Candidate Limit: 5, NProbe: 5
```
Changing only the session variable proves which value is used:
```sql
set probe_limit = 1;
explain verbose
select id from t
order by l2_distance(v, '[0,0]')
limit 5 by rank with option 'mode=post', 'nprobe=99';
-- NProbe: 1
set probe_limit = 17;
explain verbose
select id from t
order by l2_distance(v, '[0,0]')
limit 5 by rank with option 'mode=post', 'nprobe=1';
-- NProbe: 17
```
The behavior is identical on both CNs and is deterministic across repeated planning.
### Code path
The parser stores every option in `tree.RankOption.Option`, but `parseRankOption` in `pkg/sql/plan/query_builder.go` extracts only `mode` and returns a protobuf `RankOption` containing only that field. `prepareIvfIndexContext` then resolves NProbe exclusively from `probe_limit`.
The original LIMIT BY RANK request (#22919) explicitly proposed `fudge_factor` and `nprobe` as query-level controls and was closed as supported. Current parser tests also accept these option names, so silently ignoring them is misleading.
### Expected behavior
Either:
1. apply supported query-level `nprobe` / `fudge_factor` values to the vector search plan with validation and documented precedence over session defaults; or
2. reject unsupported rank options explicitly instead of accepting and discarding them.
Unknown option names should likewise not be silently treated as effective settings.
### Impact
Per-query IVF recall/latency tuning does not work. Operators may believe a query uses a larger probe budget while it continues to run with the session default, which can produce unexpected recall and latency characteristics.
Contributor guide
Assessment
This issue has not been assessed yet.