matrixorigin / matrixorigin/matrixone
[Bug]: ALTER REINDEX quantization leaves IVFFLAT entry type unchanged and breaks indexed queries
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
Changing an existing IVFFLAT index from unquantized storage to `float16`, `bf16`, `int8`, or `uint8` with `ALTER REINDEX ... QUANTIZATION` updates the logical index parameters but leaves the hidden entries column at its original `vecf32` type. Every query that uses the rebuilt index then errors with `vector dimension not matched`.
## Environment
- Branch: `main`
- Commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Deployment: local shared-storage cluster with 1 TN and 2 CNs
## Steps to reproduce
```sql
create database ivf_reindex_quant_repro;
use ivf_reindex_quant_repro;
set experimental_ivf_index = 1;
set probe_limit = 4;
create table t(id bigint primary key, v vecf32(3) not null);
insert into t
select result, concat('[', result % 31, ',', result % 37, ',', result % 41, ']')
from generate_series(1, 256) g;
create index ix using ivfflat on t(v)
lists = 4 op_type 'vector_l2_ops';
alter table t alter reindex ix ivfflat quantization 'float16';
show create table t;
select id
from t
order by l2_distance(v, '[7,7,7]')
limit 3 by rank with option 'mode=post';
```
The same behavior occurs with `quantization 'bf16'`, `'int8'`, and `'uint8'`.
## Actual behavior
`SHOW CREATE TABLE` reports the new quantization:
```sql
KEY `ix` USING ivfflat (`v`) lists = 4 op_type 'vector_l2_ops' quantization 'float16'
```
The hidden entries table still contains:
```sql
`__mo_index_centroid_fk_entry` vecf32(3) DEFAULT NULL
```
The index population path casts entries according to the new parameter, and any indexed query using `mode=pre`, `post`, `include`, or `auto` returns:
```text
ERROR 20101 (HY000): internal error: vector dimension not matched
```
## Expected behavior
Changing IVFFLAT quantization during REINDEX must rebuild the hidden entries table with the corresponding physical type (`vecf16`, `vecbf16`, `vecint8`, or `vecuint8`) so the rebuilt index remains queryable. If changing the storage type is intentionally unsupported, the ALTER statement should be rejected atomically without changing catalog metadata or index contents.
## Stability and controls
- Reproducer: `3/3` on each of two CNs for all four quantizations and all indexed modes (`pre`, `post`, `include`, `auto`).
- Fresh-index control: creating IVFFLAT directly with each quantization is queryable in all five modes.
- Full-scan control: `mode=force` remains queryable after REINDEX for all four quantizations, confirming that source rows are intact and the failure is confined to the rebuilt index path.
- Metadata control: `mo_indexes.algo_params` and `SHOW CREATE TABLE` contain the new quantization, while the hidden entries schema remains `vecf32(3)`.
## Evidence
Observed physical types:
| Quantization | Fresh CREATE entries type | ALTER REINDEX entries type |
|---|---|---|
| `float16` | `vecf16(3)` | `vecf32(3)` |
| `bf16` | `vecbf16(3)` | `vecf32(3)` |
| `int8` | `vecint8(3)` | `vecf32(3)` |
| `uint8` | `vecuint8(3)` | `vecf32(3)` |
## Code analysis
- `pkg/vectorindex/ivfflat/plugin/plan/schema.go:294-330` chooses the hidden entries column type from `QUANTIZATION`, but this schema builder is used when the hidden tables are created.
- `pkg/sql/compile/ddl.go:1215-1255` merges and persists the new quantization during `ALTER REINDEX`.
- `pkg/vectorindex/ivfflat/plugin/compile/compile.go:194-201` only calls `BuildIndexTable` when `IndexInfo()` is present; the REINDEX path reuses existing hidden tables.
- `pkg/vectorindex/ivfflat/plugin/compile/compile.go:486-503` casts rebuilt entries according to the newly persisted quantization. This produces a logical/physical type mismatch because the existing entries column remains `vecf32`.
## Regression coverage
Add an IVFFLAT ALTER REINDEX regression that changes an unquantized `vecf32` index independently to `float16`, `bf16`, `int8`, and `uint8`, verifies the hidden entry type, runs `pre/post/include/auto`, and verifies that an invalid transition leaves the old index queryable.
## Related
- Related but not duplicate: #27732 covered fresh quantized IVFFLAT DML writing incompatible entry values. This issue occurs immediately after changing quantization through `ALTER REINDEX`, before any subsequent DML, because the hidden table schema itself is not rebuilt.
- Exploration source: #28943.
Contributor guide
Assessment
This issue has not been assessed yet.