matrixorigin / matrixorigin/matrixone
[Bug]: multi-CN HNSW query CN serves stale model after async generation update
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Is there an existing issue for the same bug?
- [x] Yes. The observable BVT symptom is tracked by #27629; this issue tracks the production-side cache invalidation root cause separately.
### Branch Name
main
### Commit ID
5bc051abc44f34ccb6813e2a0bb5585d5a1db67e
### Other Environment Information
- Hardware parameters: GitHub Actions hosted Ubuntu x64 runner
- OS type: Ubuntu 22.04.5 LTS
- Deployment: multi-CN Docker Compose with Proxy
- Index: asynchronous HNSW, VECF32
### Summary
After an HNSW generation is updated through asynchronous CDC on one CN, another CN that has already loaded the previous generation can continue serving the stale in-memory model. The metadata checksum changes and the CDC consumer processes the complete delta, but the query CN does not converge to the new generation within the BVT deadline.
### Actual Behavior
In `test/distributed/cases/pessimistic_transaction/vector/vector_hnsw_async.sql`:
1. The first 10,000-row f32 generation is built.
2. A query on the query CN loads the first generation into the HNSW model cache.
3. A second 10,000-row CDC load is processed asynchronously. The consumer processes the 8,192-row and 1,808-row fragments and publishes a changed metadata checksum.
4. The query CN continues returning an old-generation nearest neighbor, for example `2638`, instead of the delta endpoint `19999` for the full 120-second wait window.
Evidence:
- Direct BVT symptom: #27629, https://github.com/matrixorigin/matrixone/issues/27629
- Failing job: https://github.com/matrixorigin/matrixone/actions/runs/32930733336/job/98063525521
- Earlier reproduction: https://github.com/matrixorigin/matrixone/actions/runs/32926456273/job/98051500919?pr=27621
- Historical related failures: #22794 and #22792
The old test implementation explicitly documented the multi-CN limitation: the model is cached in memory and may not be refreshed after CDC synchronization if a query has already loaded it.
### Expected Behavior
When a newer HNSW generation becomes visible through the shared metadata/index storage, every CN must either:
- invalidate or replace its cached older generation before serving the next indexed query; or
- reject/defer the stale cache and load the published generation within a bounded convergence interval.
A query issued after generation publication must be able to return rows introduced by that generation, including `19999` in the reproduction above.
### Impact
This is a correctness issue for distributed ANN queries, not only a performance or recall fluctuation. A CN can return results from an older data snapshot after the index update has been published, causing newly inserted rows to be invisible and nearest-neighbor results to be wrong until cache refresh or CN restart.
### Current Workaround
- Do not issue the first f32 HNSW query until the asynchronous delta generation is complete; this is the BVT-side workaround in #27631.
- For exact-result checks, force a full scan with `BY RANK WITH OPTION mode=force` where supported.
- Restarting or routing to a CN without the old cache may clear the symptom, but is not an acceptable production workaround.
### Suggested Investigation
- Make the HNSW cache key include the published metadata generation/checksum and compare it on every indexed query.
- Trace cache ownership and invalidation across CN-local consumers, model loaders, and query execution.
- Ensure old model handles cannot remain reachable after a newer generation is published.
- Add a native multi-CN integration test that warms the query CN cache, publishes CDC on another CN, and verifies the query CN observes the new generation.
### Acceptance Criteria
- A warmed query CN observes the new HNSW generation after CDC publication on a different CN.
- The test passes repeatedly with two or more CNs and with parallel 8,192/1,808-row delta fragments.
- No stale result is returned after the bounded convergence/readiness signal.
- Cache replacement is generation-safe: no stale handle, double-close, leak, or unbounded model retention.
- Existing f32/f64 async HNSW BVT coverage remains green.
Contributor guide
Assessment
This issue has not been assessed yet.