matrixorigin / matrixorigin/matrixone

[Bug]: IVFFLAT PRE and AUTO exact membership returns empty after entries persistence or physical copy

Open
#29,042 15 comments 0 reactions 0 assignees View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Problem

After `ALTER TABLE ... ADD COLUMN` rebuilds a table that has an IVFFLAT index,
an explicit `mode=pre` vector query returns an empty result even though matching
rows remain in the base table and in the vector index.

The wrong result is not tied to an old prepared plan. It also occurs with a
newly parsed static query and with a PreparedStatement created after the ALTER.
It affects both `vector_l2_ops`/`l2_distance` and
`vector_l2sq_ops`/`l2_distance_sq`, with and without IVFFLAT `INCLUDE` columns.

## Environment

- MatrixOne commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Version: `8.0.30-MatrixOne-v1.3.0`
- Topology: 2 CN / 1 TN / 1 LogService
- SQL endpoints: both CNs were checked independently

## Minimal reproduction

```sql
SET experimental_ivf_index = 1;
SET probe_limit = 1;

CREATE DATABASE ivf_copy_alter_pre;
USE ivf_copy_alter_pre;

CREATE TABLE t (
id INT PRIMARY KEY,
grp INT,
v VECF32(2)
);

INSERT INTO t VALUES
(1, 1, '[0,0]'),
(2, 1, '[1,0]'),
(3, 2, '[0,1]'),
(4, 2, '[2,0]');

CREATE INDEX ix USING IVFFLAT ON t(v)
LISTS = 1 OP_TYPE 'vector_l2_ops' INCLUDE(grp);

SELECT id FROM t
WHERE grp = 1
ORDER BY l2_distance(v, '[0,0]')
LIMIT 2 BY RANK WITH OPTION 'mode=pre';
-- 1, 2

ALTER TABLE t ADD COLUMN note INT DEFAULT 7;

SELECT id FROM t
WHERE grp = 1
ORDER BY l2_distance(v, '[0,0]')
LIMIT 2 BY RANK WITH OPTION 'mode=pre';
-- empty result
```

## Actual result

After the ALTER, `mode=pre` returns zero rows.

The post-ALTER plan still chooses the membership-filter topology:

```text
Table Scan on t (grp = 1)
-> Vector Index Scan on ix
Runtime Filter Probe
-> SEMI JOIN
Runtime Filter Build
Table Scan on t (grp = 1)
```

The base-table producer has matching primary keys, but none survive the runtime
membership filter at the IVFFLAT scan.

## Expected result

`mode=pre` should continue to return IDs `1, 2` after an unrelated column is
added.

At the same post-ALTER state, the following controls all return IDs `1, 2`:

- `mode=force`
- `mode=post`
- `mode=include` when `grp` is an INCLUDE column
- `mode=auto` in the four-row reproduction

## Reproducibility and scope

- 3 independent database recreations
- 2 CN endpoints per recreation
- 6/6 minimal observations returned the same empty `mode=pre` result
- A broader matrix reproduced the same behavior for L2 and L2SQ, static SQL,
pre-ALTER PreparedStatements, and post-ALTER PreparedStatements
- The same queries return the expected rows before the ALTER

This is a correctness issue in the explicit pre-filter path after COPY ALTER,
not approximate-nearest-neighbor recall variance: `LISTS=1` and
`probe_limit=1` scan the only IVF list, and the control paths return both rows.

Contributor guide

Open the contributing guide

Research direction

No source file or test entry point is named. Start by running the provided SQL reproduction with IVFFLAT, then trace the explicit mode=pre membership-filter path after ALTER TABLE ... ADD COLUMN and compare it with the mode=force, mode=post, and mode=include controls. Done means mode=pre returns IDs 1 and 2 after the ALTER for the documented L2 and L2SQ cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.