matrixorigin / matrixorigin/matrixone

[Bug]: IVFFLAT mode=auto returns a partial page when POST under-fills LIMIT

Open
#28,991 7 comments 0 reactions 1 assignee Claimed by @ck89119 View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

IVFFLAT `mode=auto` retries with PRE only when POST returns zero rows. If POST returns a non-empty but under-filled page, AUTO exposes the partial page even when many additional qualifying rows exist.

This violates the adaptive-mode contract in #23580, which requires AUTO to fall back when POST returns fewer rows than `LIMIT`.

## Environment

- Branch: `main`
- Commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Deployment: local 1 Log / 1 TN / 2 CN, four independent `mo-service` processes

## Steps to reproduce

```sql
SET experimental_ivf_index=1;
DROP DATABASE IF EXISTS ivf_auto_partial;
CREATE DATABASE ivf_auto_partial;
USE ivf_auto_partial;

CREATE TABLE t(
id BIGINT PRIMARY KEY,
partial_keep INT NOT NULL,
v VECF32(2)
);

INSERT INTO t
SELECT result,
IF(result >= 45, 1, 0),
CAST(CONCAT('[', result, ',0]') AS VECF32(2))
FROM generate_series(1, 1000, 1) g;

CREATE INDEX ix USING ivfflat ON t(v)
LISTS=1 OP_TYPE 'vector_l2_ops';
SET probe_limit=1;

SELECT id
FROM t
WHERE partial_keep=1
ORDER BY l2_distance(v,'[0,0]')
LIMIT 10 BY RANK WITH OPTION 'mode=auto';
```

Controls:

```sql
-- Same query with mode=post.
... LIMIT 10 BY RANK WITH OPTION 'mode=post';

-- Correct fallback/result paths.
... LIMIT 10 BY RANK WITH OPTION 'mode=pre';
... LIMIT 10 BY RANK WITH OPTION 'mode=force';
```

## Actual behavior

AUTO returns only six rows:

```text
45,46,47,48,49,50
```

POST returns the same six rows. AUTO treats this non-empty result as final and does not retry with PRE.

## Expected behavior

AUTO must return a complete 10-row page:

```text
45,46,47,48,49,50,51,52,53,54
```

PRE and force already return this result. There are 956 qualifying rows, so the six-row result is not caused by exhausting the filtered relation.

## Stability and controls

- Direct AUTO query: 3/3 on CN1 and 3/3 on CN2 returned exactly six rows.
- SQL prepared AUTO query with dynamic filter value, vector and `LIMIT`: both CNs returned the same six rows.
- `ANALYZE TABLE t` completed successfully; AUTO still returned six rows on both CNs.
- POST control: six rows, proving the result is the initial POST page.
- PRE and force controls: 10 rows, IDs `45..54`.
- A more selective predicate that leaves zero POST rows triggers AUTO fallback and correctly returns 10 rows; the defect is specific to non-zero under-fill.
- Statements are read-only and leave table/index state unchanged.

## Evidence

The two-CN 3/3 result matrix, prepared execution, post-ANALYZE result, and PRE/force controls are retained in the local exploration evidence for this commit.

## Code analysis

`pkg/sql/colexec/output/output.go` triggers `ErrVectorNeedRetryWithPreMode` only when final `rowCount == 0`. The source comment explicitly excludes `0 < rowCount < LIMIT` because partial results are not merged with retry results.

AUTO does not need to merge the POST rows with PRE rows: the retry framework already discards the failed attempt and reruns the statement. Therefore a non-empty under-filled page is currently accepted even though the PRE retry can produce the requested page.

## Regression coverage

Add a deterministic AUTO fixture where the POST path returns `0 < N < LIMIT` while at least `LIMIT` qualifying rows exist. Cover literal and prepared limits, analyzed and unanalyzed tables, and both CN endpoints. Assert equality with PRE/force row count and IDs.

## Related

- #23580 / PR #23650 define AUTO fallback when POST returns fewer results than requested.
- #28990 tracks AUTO retry failures when the vector query is nested in a CTE/derived table.
- #28988 tracks fixed-budget under-fill for HNSW, which has no IVFFLAT AUTO retry path.
- Source exploration: #28943

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.