matrixorigin / matrixorigin/matrixone
[Bug]: IVFFLAT PRE and AUTO exact membership returns empty after entries persistence or physical copy
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Problem
After `ALTER TABLE ... ADD COLUMN` rebuilds a table that has an IVFFLAT index,
an explicit `mode=pre` vector query returns an empty result even though matching
rows remain in the base table and in the vector index.
The wrong result is not tied to an old prepared plan. It also occurs with a
newly parsed static query and with a PreparedStatement created after the ALTER.
It affects both `vector_l2_ops`/`l2_distance` and
`vector_l2sq_ops`/`l2_distance_sq`, with and without IVFFLAT `INCLUDE` columns.
## Environment
- MatrixOne commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Version: `8.0.30-MatrixOne-v1.3.0`
- Topology: 2 CN / 1 TN / 1 LogService
- SQL endpoints: both CNs were checked independently
## Minimal reproduction
```sql
SET experimental_ivf_index = 1;
SET probe_limit = 1;
CREATE DATABASE ivf_copy_alter_pre;
USE ivf_copy_alter_pre;
CREATE TABLE t (
id INT PRIMARY KEY,
grp INT,
v VECF32(2)
);
INSERT INTO t VALUES
(1, 1, '[0,0]'),
(2, 1, '[1,0]'),
(3, 2, '[0,1]'),
(4, 2, '[2,0]');
CREATE INDEX ix USING IVFFLAT ON t(v)
LISTS = 1 OP_TYPE 'vector_l2_ops' INCLUDE(grp);
SELECT id FROM t
WHERE grp = 1
ORDER BY l2_distance(v, '[0,0]')
LIMIT 2 BY RANK WITH OPTION 'mode=pre';
-- 1, 2
ALTER TABLE t ADD COLUMN note INT DEFAULT 7;
SELECT id FROM t
WHERE grp = 1
ORDER BY l2_distance(v, '[0,0]')
LIMIT 2 BY RANK WITH OPTION 'mode=pre';
-- empty result
```
## Actual result
After the ALTER, `mode=pre` returns zero rows.
The post-ALTER plan still chooses the membership-filter topology:
```text
Table Scan on t (grp = 1)
-> Vector Index Scan on ix
Runtime Filter Probe
-> SEMI JOIN
Runtime Filter Build
Table Scan on t (grp = 1)
```
The base-table producer has matching primary keys, but none survive the runtime
membership filter at the IVFFLAT scan.
## Expected result
`mode=pre` should continue to return IDs `1, 2` after an unrelated column is
added.
At the same post-ALTER state, the following controls all return IDs `1, 2`:
- `mode=force`
- `mode=post`
- `mode=include` when `grp` is an INCLUDE column
- `mode=auto` in the four-row reproduction
## Reproducibility and scope
- 3 independent database recreations
- 2 CN endpoints per recreation
- 6/6 minimal observations returned the same empty `mode=pre` result
- A broader matrix reproduced the same behavior for L2 and L2SQ, static SQL,
pre-ALTER PreparedStatements, and post-ALTER PreparedStatements
- The same queries return the expected rows before the ALTER
This is a correctness issue in the explicit pre-filter path after COPY ALTER,
not approximate-nearest-neighbor recall variance: `LISTS=1` and
`probe_limit=1` scan the only IVF list, and the control paths return both rows.
Contributor guide
Research direction
No source file or test entry point is named. Start by running the provided SQL reproduction with IVFFLAT, then trace the explicit mode=pre membership-filter path after ALTER TABLE ... ADD COLUMN and compare it with the mode=force, mode=post, and mode=include controls. Done means mode=pre returns IDs 1 and 2 after the ALTER for the documented L2 and L2SQ cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100