matrixorigin / matrixorigin/matrixone
[Bug]: async IVFFLAT INCLUDE cannot maintain JSON strings containing single quotes
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Environment
- MatrixOne commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Deployment: local shared-TN cluster with two CNs
- Scope: asynchronous IVFFLAT INCLUDE maintenance through ISCP
## Problem
An asynchronous IVFFLAT index cannot populate or maintain an included JSON
column when a JSON string value contains a single quote. The source value is
valid JSON, but the entries table remains empty because ISCP embeds the JSON
text in a SQL single-quoted literal without escaping it.
Safe JSON, JSON containing an escaped backslash, and JSON containing an escaped
newline complete on the same cluster. A synchronous IVFFLAT build stores the
quoted JSON value correctly.
## Minimal reproduction
```sql
set experimental_ivf_index=1;
create database ivf_async_include_json;
use ivf_async_include_json;
create table t(
id bigint primary key,
j json,
v vecf32(2)
);
insert into t
select result,
json_object('s', 'O''Reilly'),
cast(concat('[',result,',0]') as vecf32(2))
from generate_series(1,10) g;
create index ix using ivfflat on t(v)
lists=1
op_type 'vector_l2_ops'
quantization 'float16'
include(j)
async;
```
After waiting well beyond the time required by safe controls:
```sql
select count(*) from ;
-- 0 (expected 10)
```
## Reproduction matrix
| JSON value | async entries | synchronous entries |
|---|---:|---:|
| `{"s":"O'Reilly"}` | 0 / 10 | 10 / 10 |
| `{"s":"safe"}` | 10 / 10 | 10 / 10 |
| JSON string containing backslash | 10 / 10 | 10 / 10 |
| JSON string containing newline | 10 / 10 | 10 / 10 |
Three independently created async databases containing `O'Reilly` remained at
zero entries. The safe controls reached 10 entries in about 14 seconds.
`ALTER TABLE ... ALTER REINDEX ... FORCE_SYNC` builds all 10 entries and stores
`{"s":"O'Reilly"}` correctly. Inserting another quoted JSON value afterward
leaves the entries table at 10 rows and the new primary key absent, so ongoing
async maintenance is affected as well as initial replay.
## Root cause
The JSON branch of `pkg/iscp/util.go:convertColIntoSql` writes
`bytejson.ByteJson.String()` directly between SQL single quotes:
```go
sqlBuff = appendByte(sqlBuff, '\'')
sqlBuff = appendString(sqlBuff, data.(bytejson.ByteJson).String())
sqlBuff = appendByte(sqlBuff, '\'')
```
It does not escape embedded single quotes. The text/string branches perform
quote and backslash escaping, but JSON does not use that path. The generated
ISCP SQL therefore terminates the literal at the JSON string's quote.
## Expected behavior
Every valid JSON value must be copied unchanged into the IVFFLAT entries table
during initial async replay and later INSERT/UPDATE operations. JSON SQL
serialization must escape the complete textual representation or use a
parameter/binary-safe representation.
## Regression coverage
- JSON strings containing single quote, backslash, newline, tab, and Unicode;
- nested objects and arrays containing those strings;
- initial replay plus later INSERT/UPDATE;
- synchronous build control;
- round-trip comparison with `JSON_EXTRACT` and the source JSON value.
Contributor guide
Assessment
This issue has not been assessed yet.