matrixorigin / matrixorigin/matrixone
[Bug] IVF pre-filter 向量查询 EXPLAIN ANALYZE 导致 CN OOMKilled (10M 向量, 100 并发)
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## 环境
| Field | Value |
|-------|-------|
| 环境 | TKE (3CN + 1DN + 2Proxy) |
| 分支 | 4.2-dev |
| Commit | `fa6c57cbf` |
| Run | https://github.com/matrixorigin/mo-nightly-regression/actions/runs/30932449415/job/92165620013 |
| Namespace | `mo-branch-commit-fa6c57cbf-20260804` |
## 问题
IVF pre-filter 向量查询(`wiki_recall_l2_only_pre`)在 10M 向量表上执行 EXPLAIN ANALYZE(100 并发),CN pod `tp-cn-6gftc` 被 K8s OOMKilled。
### 时间线
```
01:29:28 EXPLAIN ANALYZE 开始 (wiki_recall_l2_only_pre, IVF pre-filter, 100 并发)
01:29:49 CN pod tp-cn-6gftc OOMKilled (exit 137)
01:29:52 客户端收到 ERROR 20503: rpc stream closed
```
### Pod 重启记录
```
Pod: nightly-regression-dis-tp-cn-6gftc
Exit code: 137 (SIGKILL)
Reason: OOMKilled
Time: 2026-08-05T01:29:49Z
```
### 配置
- `cn_limits_memory: 55Gi`(pod hard limit)
- `cn_gomemlimit: 25000MiB`(Go soft memory limit)
- Go GOMEMLIMIT 是 soft limit(25GB 触发 GC),但 pod limit 55GB 是 hard limit
- 实际内存超过 55GB,被 K8s OOM Killer 杀掉
### 查询详情
```sql
-- wiki_recall_l2_only_pre: IVF pre-filter, L2 distance
-- Table: historical_file_blocks_wiki_ivfflat10m (10M vectors)
-- Concurrency: 100
-- Queries: 10000
-- Filter mode: pre
-- Probe: 5
-- Latency stats: avg=6066ms, p50=4342ms, p99=30332ms, max=201293ms
```
### 影响
- CN OOMKilled 后重启,重启期间所有查询受影响
- QPS 从 28.94(上次)降到 16.38(-43.4%)
- Recall: 0.2178
### 原因
IVF pre-filter 模式下,100 并发查询 10M 向量表,内存用量超过 pod limit 55Gi。pre-filter 需要先过滤再搜索,计算和内存开销大。单次查询 max 延迟 201 秒,说明有极端慢查询占用大量内存。
Contributor guide
Assessment
This issue has not been assessed yet.