matrixorigin / matrixorigin/matrixone
[Bug]: async IVFFLAT with int8/uint8 quantization and INCLUDE never populates entries
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Environment
- MatrixOne commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Deployment: local shared-TN cluster with two CNs
- Scope: asynchronous IVFFLAT maintenance through ISCP
## Problem
An asynchronous IVFFLAT index that combines affine narrow quantization
(`int8` or `uint8`) with any `INCLUDE` column never populates its entries
table. The metadata and centroid tables complete, but the entries table stays
empty indefinitely, so all index-using modes return an empty result.
The same table and workload complete when either feature is removed. They also
complete with `float16` or `bf16` quantization plus `INCLUDE`.
## Minimal reproduction
```sql
set experimental_ivf_index=1;
create database ivf_async_affine_include;
use ivf_async_affine_include;
create table t(
id bigint primary key,
cat int,
v vecf32(2)
);
insert into t
select result, result % 3,
cast(concat('[', result, ',0]') as vecf32(2))
from generate_series(1, 20) g;
create index ix using ivfflat on t(v)
lists=2
op_type 'vector_l2_ops'
quantization 'int8'
include(cat)
async;
```
After waiting 30-40 seconds, the metadata table contains its version and
quantization rows and the centroid table contains two centroids, but:
```sql
select count(*) from ;
-- 0 (expected 20)
```
The same result occurs with `quantization 'uint8'`.
## Reproduction matrix
The matrix was run repeatedly with independent databases on the same cluster:
| Configuration | Entries after 30-40 s |
|---|---:|
| `int8 + INCLUDE(int)` | 0 / 20 |
| `int8 + INCLUDE(varchar)` | 0 / 20 |
| `uint8 + INCLUDE(int)` | 0 / 20 |
| `float16 + INCLUDE(int)` | 20 / 20 |
| `bf16 + INCLUDE(int)` | 20 / 20 |
| `int8` without INCLUDE | 100 / 100 |
| INCLUDE without quantization | 100 / 100 |
Three independent `int8 + INCLUDE` databases remained at 0 entries while the
controls reached their full row count in about 12-14 seconds.
For the empty `uint8 + INCLUDE` index, both CNs returned:
| mode | result for nearest 3 rows around `[10,0]` |
|---|---|
| `force` | `9,10,11` |
| `pre` | empty |
| `post` | empty |
| `include` | empty |
| `auto` | empty |
A `FORCE_SYNC` reindex builds all entries synchronously, but later asynchronous
delete/insert maintenance still does not move: the deleted key remains and the
inserted key never appears.
## Root cause
The ISCP IVFFLAT writer builds its source-column order as:
```
primary key, vector part(s), include column(s)
```
However, `toIvfflatUpsert` assumes the last source column is always the vector
and applies the `int8`/`uint8` quantization expression to that last column. Once
an INCLUDE column is present, the last source column is an INCLUDE value. The
generated projection therefore quantizes the wrong value and the batch cannot
be written to the entries table.
Relevant code:
- `pkg/iscp/index_sqlwriter.go:639-647` appends vector parts and then INCLUDE columns.
- `pkg/iscp/index_sqlwriter.go:741-768` quantizes and replaces `cnames[len(cnames)-1]`.
- The vector used by `CENTROIDX` is correctly referenced as `cnames[1]` at line 805, confirming the projection index should also refer to the vector position rather than the final column.
The synchronous full-build path is not affected because it constructs the
entry expression directly from the indexed vector column before appending
INCLUDE projections (`pkg/vectorindex/ivfflat/plugin/compile/compile.go`).
## Expected behavior
Async `int8`/`uint8` IVFFLAT indexes with INCLUDE columns should populate all
existing rows and maintain later inserts, updates, and deletes. The quantization
expression must be applied to the indexed vector source column while INCLUDE
columns retain their original values.
## Regression coverage
Add ISCP writer and end-to-end cases for:
- `int8` and `uint8` with one and multiple INCLUDE columns;
- nullable and non-null INCLUDE values of different supported types;
- initial async replay and later insert/update/delete batches;
- generated SQL column order: PK, quantized vector, INCLUDE values;
- `float16`/`bf16` controls to keep the non-affine path covered.
Contributor guide
Assessment
This issue has not been assessed yet.