matrixorigin / matrixorigin/matrixone

[Bug]: ALTER REINDEX quantization leaves IVFFLAT entry type unchanged and breaks indexed queries

Open
#29,009 1 comment 0 reactions 1 assignee Claimed by @cpegeric View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

Changing an existing IVFFLAT index from unquantized storage to `float16`, `bf16`, `int8`, or `uint8` with `ALTER REINDEX ... QUANTIZATION` updates the logical index parameters but leaves the hidden entries column at its original `vecf32` type. Every query that uses the rebuilt index then errors with `vector dimension not matched`.

## Environment

- Branch: `main`
- Commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Deployment: local shared-storage cluster with 1 TN and 2 CNs

## Steps to reproduce

```sql
create database ivf_reindex_quant_repro;
use ivf_reindex_quant_repro;
set experimental_ivf_index = 1;
set probe_limit = 4;

create table t(id bigint primary key, v vecf32(3) not null);
insert into t
select result, concat('[', result % 31, ',', result % 37, ',', result % 41, ']')
from generate_series(1, 256) g;

create index ix using ivfflat on t(v)
lists = 4 op_type 'vector_l2_ops';

alter table t alter reindex ix ivfflat quantization 'float16';

show create table t;
select id
from t
order by l2_distance(v, '[7,7,7]')
limit 3 by rank with option 'mode=post';
```

The same behavior occurs with `quantization 'bf16'`, `'int8'`, and `'uint8'`.

## Actual behavior

`SHOW CREATE TABLE` reports the new quantization:

```sql
KEY `ix` USING ivfflat (`v`) lists = 4 op_type 'vector_l2_ops' quantization 'float16'
```

The hidden entries table still contains:

```sql
`__mo_index_centroid_fk_entry` vecf32(3) DEFAULT NULL
```

The index population path casts entries according to the new parameter, and any indexed query using `mode=pre`, `post`, `include`, or `auto` returns:

```text
ERROR 20101 (HY000): internal error: vector dimension not matched
```

## Expected behavior

Changing IVFFLAT quantization during REINDEX must rebuild the hidden entries table with the corresponding physical type (`vecf16`, `vecbf16`, `vecint8`, or `vecuint8`) so the rebuilt index remains queryable. If changing the storage type is intentionally unsupported, the ALTER statement should be rejected atomically without changing catalog metadata or index contents.

## Stability and controls

- Reproducer: `3/3` on each of two CNs for all four quantizations and all indexed modes (`pre`, `post`, `include`, `auto`).
- Fresh-index control: creating IVFFLAT directly with each quantization is queryable in all five modes.
- Full-scan control: `mode=force` remains queryable after REINDEX for all four quantizations, confirming that source rows are intact and the failure is confined to the rebuilt index path.
- Metadata control: `mo_indexes.algo_params` and `SHOW CREATE TABLE` contain the new quantization, while the hidden entries schema remains `vecf32(3)`.

## Evidence

Observed physical types:

| Quantization | Fresh CREATE entries type | ALTER REINDEX entries type |
|---|---|---|
| `float16` | `vecf16(3)` | `vecf32(3)` |
| `bf16` | `vecbf16(3)` | `vecf32(3)` |
| `int8` | `vecint8(3)` | `vecf32(3)` |
| `uint8` | `vecuint8(3)` | `vecf32(3)` |

## Code analysis

- `pkg/vectorindex/ivfflat/plugin/plan/schema.go:294-330` chooses the hidden entries column type from `QUANTIZATION`, but this schema builder is used when the hidden tables are created.
- `pkg/sql/compile/ddl.go:1215-1255` merges and persists the new quantization during `ALTER REINDEX`.
- `pkg/vectorindex/ivfflat/plugin/compile/compile.go:194-201` only calls `BuildIndexTable` when `IndexInfo()` is present; the REINDEX path reuses existing hidden tables.
- `pkg/vectorindex/ivfflat/plugin/compile/compile.go:486-503` casts rebuilt entries according to the newly persisted quantization. This produces a logical/physical type mismatch because the existing entries column remains `vecf32`.

## Regression coverage

Add an IVFFLAT ALTER REINDEX regression that changes an unquantized `vecf32` index independently to `float16`, `bf16`, `int8`, and `uint8`, verifies the hidden entry type, runs `pre/post/include/auto`, and verifies that an invalid transition leaves the old index queryable.

## Related

- Related but not duplicate: #27732 covered fresh quantized IVFFLAT DML writing incompatible entry values. This issue occurs immediately after changing quantization through `ALTER REINDEX`, before any subsequent DML, because the hidden table schema itself is not rebuilt.
- Exploration source: #28943.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.