matrixorigin / matrixorigin/matrixone

[Bug]: multi-CN HNSW query CN serves stale model after async generation update

Open
#27,632 5 comments 0 reactions 1 assignee Claimed by @XuPeng-SH View on GitHub
Bug fix severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] Yes. The observable BVT symptom is tracked by #27629; this issue tracks the production-side cache invalidation root cause separately.

### Branch Name

main

### Commit ID

5bc051abc44f34ccb6813e2a0bb5585d5a1db67e

### Other Environment Information

- Hardware parameters: GitHub Actions hosted Ubuntu x64 runner
- OS type: Ubuntu 22.04.5 LTS
- Deployment: multi-CN Docker Compose with Proxy
- Index: asynchronous HNSW, VECF32

### Summary

After an HNSW generation is updated through asynchronous CDC on one CN, another CN that has already loaded the previous generation can continue serving the stale in-memory model. The metadata checksum changes and the CDC consumer processes the complete delta, but the query CN does not converge to the new generation within the BVT deadline.

### Actual Behavior

In `test/distributed/cases/pessimistic_transaction/vector/vector_hnsw_async.sql`:

1. The first 10,000-row f32 generation is built.
2. A query on the query CN loads the first generation into the HNSW model cache.
3. A second 10,000-row CDC load is processed asynchronously. The consumer processes the 8,192-row and 1,808-row fragments and publishes a changed metadata checksum.
4. The query CN continues returning an old-generation nearest neighbor, for example `2638`, instead of the delta endpoint `19999` for the full 120-second wait window.

Evidence:

- Direct BVT symptom: #27629, https://github.com/matrixorigin/matrixone/issues/27629
- Failing job: https://github.com/matrixorigin/matrixone/actions/runs/32930733336/job/98063525521
- Earlier reproduction: https://github.com/matrixorigin/matrixone/actions/runs/32926456273/job/98051500919?pr=27621
- Historical related failures: #22794 and #22792

The old test implementation explicitly documented the multi-CN limitation: the model is cached in memory and may not be refreshed after CDC synchronization if a query has already loaded it.

### Expected Behavior

When a newer HNSW generation becomes visible through the shared metadata/index storage, every CN must either:

- invalidate or replace its cached older generation before serving the next indexed query; or
- reject/defer the stale cache and load the published generation within a bounded convergence interval.

A query issued after generation publication must be able to return rows introduced by that generation, including `19999` in the reproduction above.

### Impact

This is a correctness issue for distributed ANN queries, not only a performance or recall fluctuation. A CN can return results from an older data snapshot after the index update has been published, causing newly inserted rows to be invisible and nearest-neighbor results to be wrong until cache refresh or CN restart.

### Current Workaround

- Do not issue the first f32 HNSW query until the asynchronous delta generation is complete; this is the BVT-side workaround in #27631.
- For exact-result checks, force a full scan with `BY RANK WITH OPTION mode=force` where supported.
- Restarting or routing to a CN without the old cache may clear the symptom, but is not an acceptable production workaround.

### Suggested Investigation

- Make the HNSW cache key include the published metadata generation/checksum and compare it on every indexed query.
- Trace cache ownership and invalidation across CN-local consumers, model loaders, and query execution.
- Ensure old model handles cannot remain reachable after a newer generation is published.
- Add a native multi-CN integration test that warms the query CN cache, publishes CDC on another CN, and verifies the query CN observes the new generation.

### Acceptance Criteria

- A warmed query CN observes the new HNSW generation after CDC publication on a different CN.
- The test passes repeatedly with two or more CNs and with parallel 8,192/1,808-row delta fragments.
- No stale result is returned after the bounded convergence/readiness signal.
- Cache replacement is generation-safe: no stale handle, double-close, leak, or unbounded model retention.
- Existing f32/f64 async HNSW BVT coverage remains green.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.