matrixorigin / matrixorigin/matrixone
[Bug]: early query during async IVFFLAT build pins incomplete centroid cache on one CN
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Description
An IVFFLAT query issued while an `ASYNC` index is completing its initial build can cache an empty centroid generation on one CN. After all hidden centroid and entry rows are committed, that CN continues searching only bucket `1`, while a CN that first queries after the build sees the complete index and returns the correct nearest neighbor.
This is not the accepted temporary empty-result window of an asynchronous index. The incorrect result remains after the hidden index is fully populated, and repeated reads refresh the resident cache entry.
## Environment
- Branch: `main`
- Commit: `01d60e1c4ded1b0f3fc1a4ecd75ce54e95e23b90`
- Deployment: local 1 Log / 1 TN / 2 CN, four independent `mo-service` processes
- CPU IVFFLAT, `vecf32(2)`, `lists=4`, `probe_limit=4`
## Steps to reproduce
```sql
SET experimental_ivf_index=1;
CREATE DATABASE ivf_async_partial_cache;
USE ivf_async_partial_cache;
CREATE TABLE t(id BIGINT PRIMARY KEY, v VECF32(2));
INSERT INTO t
SELECT result, CAST(CONCAT('[', result, ',0]') AS VECF32(2))
FROM generate_series(1,100) g;
CREATE INDEX ix USING ivfflat ON t(v)
LISTS=4 OP_TYPE 'vector_l2_ops' ASYNC;
```
Immediately poll the following query on CN1 while the async build runs:
```sql
SET probe_limit=4;
SELECT id FROM t
ORDER BY l2_distance(v,'[50,0]')
LIMIT 1 BY RANK WITH OPTION 'mode=include';
```
Wait until the IVFFLAT entries hidden table contains all 100 source PKs. Then execute the same query three times on CN1 and on CN2. CN2 must not have queried this index before the build completed.
## Actual behavior
Three independent databases reproduced the same split:
```text
run=1 hidden_entries=100 cn1=76,76,76 cn2=50,50,50 force=50,50,50
run=2 hidden_entries=100 cn1=76,76,76 cn2=50,50,50 force=50,50,50
run=3 hidden_entries=100 cn1=76,76,76 cn2=50,50,50 force=50,50,50
```
The final hidden-table state is complete:
- four non-NULL centroids: `[63,0]`, `[88,0]`, `[13,0]`, `[38,0]`;
- 100 entries covering PKs `1..100`;
- bucket `1` contains PKs `76..100`.
On the warmed CN1, `pre`, `post`, `include`, and `auto` all return `76` as the nearest row. On fresh CN2 all four modes return the correct row `50`, with distance zero. `mode=force` also returns `50` on CN1.
Running a synchronous `ALTER REINDEX ... FORCE_SYNC` evicts/rebuilds the local cache; CN1 then immediately returns `50` in all three independent databases.
## Expected behavior
Once the asynchronous build has committed the complete centroid/entry generation, every CN must converge to that generation. A query during the not-ready interval must not pin a partial routing model under the final cache key.
It is acceptable for an async index to have a temporary not-ready interval, but it is not acceptable for one CN to keep returning a different nearest-neighbor set after the persisted index is complete.
## Code analysis
The async create path writes metadata version `0` and defers `ivf_create` to the CDC pipeline (`pkg/vectorindex/ivfflat/plugin/compile/compile.go`). Queries cache centroid routing under `:` (`pkg/vectorindex/ivfflat/plan_reader.go`).
During the build window:
1. `IvfflatSearchIndex.LoadCentroids` returns success when the centroid scan has no batches or zero non-NULL centroids, leaving `idx.Centroids == nil` (`pkg/vectorindex/ivfflat/search.go`).
2. The cache therefore publishes a successful IVFFLAT generation under `centroids-table:0`.
3. `rankCentroids` treats `Centroids == nil` as the empty-index sentinel and routes every query to centroid id `1`.
4. The async build later fills all centroids and entries without changing metadata version `0`. IVFFLAT does not report this as an empty generation to `VectorIndexCache`, and it has no freshness check for this transition, so the warmed CN retains the sentinel route.
The general cache already has an empty-generation eviction hook, but IVFFLAT does not set it. The observed wrong row `76` is the first PK in bucket `1`, matching the sentinel route exactly.
## Regression coverage
Add an end-to-end two-CN async-build race:
1. warm CN1 while the initial IVF centroid table is empty;
2. wait for the hidden centroid and entry tables to become complete;
3. verify warmed CN1 and fresh CN2 return the same nearest IDs for `pre`, `post`, `include`, and `auto`;
4. repeat for default and quantized entries;
5. keep the accepted immediate not-ready behavior separate from post-build cache convergence.
## Related
- #26029 was closed as the accepted immediate empty-result behavior of ASYNC indexes. This issue is different: the hidden index is complete, but a CN remains pinned to a partial routing cache.
- #28943 concerns unordered multi-CN candidate merging after both CNs have valid IVF models; this issue occurs before candidate merge because the CNs use different cached centroid routes.
Contributor guide
Assessment
This issue has not been assessed yet.