matrixorigin / matrixorigin/matrixone
[Bug]: async IVFFLAT INCLUDE cannot maintain BIT values containing quote or backslash bytes
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Environment
- MatrixOne commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Deployment: local shared-TN cluster with two CNs
- Scope: asynchronous IVFFLAT INCLUDE maintenance through ISCP
## Problem
IVFFLAT declares `BIT` as a supported INCLUDE type, but asynchronous index
maintenance cannot process BIT values whose byte representation contains a
single quote (`0x27`) or backslash (`0x5c`). The entries table remains empty for
the initial replay, or stops receiving later DML after a synchronous rebuild.
Safe BIT bytes such as `0x00`, `0x01`, and `0xff` complete in the same cluster.
A synchronous IVFFLAT build also stores `0x27` correctly, isolating the problem
to ISCP SQL serialization.
## Minimal reproduction
```sql
set experimental_ivf_index=1;
create database ivf_async_include_bit;
use ivf_async_include_bit;
create table t(
id bigint primary key,
flags bit(8),
v vecf32(2)
);
insert into t
select result,
b'00100111', -- 0x27, ASCII single quote
cast(concat('[',result,',0]') as vecf32(2))
from generate_series(1,10) g;
create index ix using ivfflat on t(v)
lists=1
op_type 'vector_l2_ops'
quantization 'float16'
include(flags)
async;
```
After waiting well beyond the time needed by safe controls:
```sql
select count(*) from ;
-- 0 (expected 10)
```
The same result occurs with `b'01011100'` (`0x5c`, backslash).
## Reproduction matrix
| INCLUDE value | async entries | synchronous entries |
|---|---:|---:|
| `BIT(8) 0x27` | 0 / 10 | 10 / 10 |
| `BIT(8) 0x5c` | 0 / 10 | 10 / 10 |
| `BIT(8) 0x00` | 10 / 10 | 10 / 10 |
| `BIT(8) 0x01` | 10 / 10 | 10 / 10 |
| `BIT(8) 0xff` | 10 / 10 | 10 / 10 |
Three independently created async databases containing `0x27` remained at zero
entries. Safe controls created at the same time reached 10 entries in about 14
seconds.
After `ALTER TABLE ... ALTER REINDEX ... FORCE_SYNC`, the affected index contains
all 10 rows and stores `HEX(flags) = '27'`. Inserting another source row with
`0x27` then leaves the entries table at 10 rows and the new primary key absent,
confirming that ongoing async maintenance is affected as well as initial replay.
## Root cause
`pkg/iscp/util.go:convertColIntoSql` serializes `BIT` by putting its raw bytes
between single quotes:
```go
sqlBuff = appendByte(sqlBuff, '\'')
sqlBuff = appendBytes(sqlBuff, b)
sqlBuff = appendByte(sqlBuff, '\'')
```
Unlike string handling, this branch does not escape quote or backslash bytes;
unlike `BINARY`/`VARBINARY`/`BLOB`, it does not use a hex literal. The generated
ISCP SQL is therefore syntactically invalid or changes escape interpretation for
these valid BIT values.
## Expected behavior
Every valid BIT value must be copied byte-for-byte into the IVFFLAT entries
table during initial async replay and later INSERT/UPDATE operations. BIT should
be serialized with a binary-safe literal representation.
## Regression coverage
- all 256 `BIT(8)` values, especially `0x00`, `0x27`, `0x5c`, and `0xff`;
- widths that are not a multiple of eight;
- one and multiple BIT INCLUDE columns;
- initial replay plus later INSERT/UPDATE;
- synchronous build control;
- generated ISCP SQL must use a binary-safe representation.
Contributor guide
Assessment
This issue has not been assessed yet.