matrixorigin / matrixorigin/matrixone

[Bug] IVF pre-filter 向量查询 EXPLAIN ANALYZE 导致 CN OOMKilled (10M 向量, 100 并发)

Open
#26,720 3 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
kind/bug
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## 环境

| Field | Value |
|-------|-------|
| 环境 | TKE (3CN + 1DN + 2Proxy) |
| 分支 | 4.2-dev |
| Commit | `fa6c57cbf` |
| Run | https://github.com/matrixorigin/mo-nightly-regression/actions/runs/30932449415/job/92165620013 |
| Namespace | `mo-branch-commit-fa6c57cbf-20260804` |

## 问题

IVF pre-filter 向量查询(`wiki_recall_l2_only_pre`)在 10M 向量表上执行 EXPLAIN ANALYZE(100 并发),CN pod `tp-cn-6gftc` 被 K8s OOMKilled。

### 时间线

```
01:29:28 EXPLAIN ANALYZE 开始 (wiki_recall_l2_only_pre, IVF pre-filter, 100 并发)
01:29:49 CN pod tp-cn-6gftc OOMKilled (exit 137)
01:29:52 客户端收到 ERROR 20503: rpc stream closed
```

### Pod 重启记录

```
Pod: nightly-regression-dis-tp-cn-6gftc
Exit code: 137 (SIGKILL)
Reason: OOMKilled
Time: 2026-08-05T01:29:49Z
```

### 配置

- `cn_limits_memory: 55Gi`(pod hard limit)
- `cn_gomemlimit: 25000MiB`(Go soft memory limit)
- Go GOMEMLIMIT 是 soft limit(25GB 触发 GC),但 pod limit 55GB 是 hard limit
- 实际内存超过 55GB,被 K8s OOM Killer 杀掉

### 查询详情

```sql
-- wiki_recall_l2_only_pre: IVF pre-filter, L2 distance
-- Table: historical_file_blocks_wiki_ivfflat10m (10M vectors)
-- Concurrency: 100
-- Queries: 10000
-- Filter mode: pre
-- Probe: 5
-- Latency stats: avg=6066ms, p50=4342ms, p99=30332ms, max=201293ms
```

### 影响

- CN OOMKilled 后重启,重启期间所有查询受影响
- QPS 从 28.94(上次)降到 16.38(-43.4%)
- Recall: 0.2178

### 原因

IVF pre-filter 模式下,100 并发查询 10M 向量表,内存用量超过 pod limit 55Gi。pre-filter 需要先过滤再搜索,计算和内存开销大。单次查询 max 延迟 201 秒,说明有极端慢查询占用大量内存。

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.