matrixorigin / matrixorigin/matrixone

[Bug]: 10M IVFFlat workload intermittently closes RPC/backend connections

Open
#27,757 10 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
kind/bug needs-triage severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Description

The 10M IVFFlat regression workload intermittently loses MatrixOne client connections under real ANN recall load. The same TKE run successfully imports the complete dataset and builds the IVFFlat index, but concurrent recall requests subsequently fail with `backend connection closed`, followed by `rpc stream closed` from `EXPLAIN ANALYZE` on the same indexed query.

This prevents the regression job from producing valid recall/latency evidence. The immediate failing component (CN, TN, RPC, or vector query path) is not yet identified.

## Environment

- Branch: `4.2-dev`
- MatrixOne commit: `03ffd507597ded14abbce44c31616e5979463096`
- Deployment: TKE distributed MatrixOne, 3 CN + 1 TN
- CN limits: 55 GiB memory; Go memory limit 25 GiB
- Dataset: 10,000,000 rows, `VECF32(768)`
- Index: `IVFFlat`, `lists=3162`, `op_type=vector_l2_ops`, `quantization=float32`
- Recall workload: 10,000 ANN ground-truth queries, `k=10`, concurrency 100, `probe_limit=5`, pre-filter

**Validation status:** This issue was observed on the `4.2-dev` commit above. MatrixOne `main` validation is pending; the next P0/P1 run will append the resolved `main` commit and outcome. Until then, this issue must not be interpreted as confirmed reproducible on `main`.

## Steps to reproduce

1. Create a 10M-row `VECF32(768)` table from the Wiki S3 regression dataset.
2. Create an IVFFlat L2 index with the parameters above and wait for index build completion.
3. Run the pre-filter L2 recall workload against the indexed table: 10,000 queries, `k=10`, concurrency 100, `probe_limit=5`.
4. Run `EXPLAIN ANALYZE` for the same indexed vector query after the recall passes.

Automated reproducer: [IVF PRE POST FILTER VECTOR RECALL TEST](https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33056956661/job/98468504442).

## Actual behavior

The S3 import completed with `affected_rows=10000000`, and IVFFlat index creation completed. During the pre-filter recall workload, the client recorded 23 occurrences of:

```text
query failed: (20502, 'backend connection closed')
```

Both 10k passes emitted partial summaries, but those metrics are invalid because requests were disconnected. The following `EXPLAIN ANALYZE` failed with:

```text
ERROR 20503 (HY000): rpc stream closed
```

The regression correctly rejected the partial quality metrics and stopped before the remaining scenarios.

## Expected behavior

All 10,000 concurrent indexed recall requests and the subsequent `EXPLAIN ANALYZE` should complete without backend/RPC closure. The job should produce complete, valid recall and latency metrics; a transient server-side failure must not be silently converted into a usable benchmark result.

## Stability / impact

- Recall-stage incident: one 10M run, with 23 client-side `backend connection closed` records plus the subsequent `rpc stream closed` failure.
- Import-stage signal: three earlier P1 workflow runs and the first import attempt of this run also ended with `ERROR 20503: rpc stream closed`. The current run's second import attempt succeeded, so these import failures are recorded as related connection-stability evidence rather than asserted to have the same root cause.
- The failure blocks the P1 IVFFlat pre/post-filter quality gate; there is currently no valid 10M recall baseline from this run.

## Evidence

- Failing TKE job: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/33056956661/job/98468504442
- Recall artifact: `wiki-all-10m-recall-commit-03ffd5075` (artifact id `9643420022`)
- Partial summaries before rejection: recall@10 `0.2294` / `0.2338`; QPS `6.17` / `35.94`. These are not valid baselines because the workload contained connection failures.
- No pod restart/OOM diagnostics were collected for the recall-stage failure; root cause remains unknown.

## Regression coverage

The failing workload is already covered by the TKE nightly regression. After root-cause remediation, rerun the same 10M IVFFlat pre-filter workload and require:

1. zero client connection/RPC closure errors;
2. both recall passes to complete with all 10,000 requests accounted for; and
3. the follow-up `EXPLAIN ANALYZE` to complete successfully.

The nightly job should additionally collect CN/TN/Log pod status, events, restart counts, and relevant logs whenever recall-stage connection errors occur, so future occurrences can be diagnosed rather than only reported by the client.

## Related

Discovered while executing the vector P1 large-data IVF pre/post-filter recall regression. This is a runtime/connection-stability failure, not a recall-threshold regression. Root cause and exact owning component require triage.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.