matrixorigin / matrixorigin/matrixone
[Bug] sysbench1000w Insert Duplicate fails: CN memory pressure (40-41 GiB) causes cascading connection failures
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Issue
sysbench1000w "Insert Duplicate 1000 threads" test fails with connection timeouts due to CN memory pressure. During the test, CN memory spikes to 40-41 GiB (close to the 55 GiB limit), triggering DN OOM alerts and cascading connection failures across the cluster.
## Environment
| Field | Value |
|---|---|
| **CI Run** | https://github.com/matrixorigin/mo-nightly-regression/actions/runs/29940401960/job/89124090085 |
| **MO Commit** | `273400583d84d8da0ea2d61d9cb1c3dbd75db708` (4.2-dev) |
| **Namespace** | `mo-branch-commit-273400583-20260723` |
| **Cluster** | TKE (3 CN + 1 DN + 2 Proxy) |
| **Failure Time** | 2026-07-23 06:24:24 - 06:31:37 UTC |
| **Failed Step** | Step 34: Start OLTP Insert Duplicate Test No Auto_increment With 1000 Threads |
## Failure Details
**Step 34** ran for 7 minutes (06:24:24 - 06:31:37 UTC) before timing out. The test normally completes in 5-7 minutes.
### Error Messages
1. **CN service error** (06:31:56 on CN `gr4jr`):
```
cannot get salt, maybe not use proxy
error: read tcp 10.143.140.190:6001->10.143.140.182:49052: i/o timeout
```
2. **Proxy connection failures** (06:31:50-55):
```
close conneciton failed
error: close tcp4 10.143.140.186:6001->10.128.1.138:42544: use of closed network connection
```
3. **DN OOM alerts** (06:31:53-59):
```
MergeExecutorEvent-PauseDueToOOMAlert
available-mem: 0
```
### Memory Usage at Failure Time
**CN Pods** (06:28-06:30 UTC peak):
| CN Pod | Peak Memory | Time |
|---|---|---|
| `7f25f` | 41.4 GiB | 06:28 |
| `gr4jr` | 40.8 GiB | 06:30 |
| `cznhr` | 40.5 GiB | 06:25 |
**DN Pod**: 38.9 GiB peak at 06:31
### Normal vs Failed Tests
| Test | Duration | Result |
|---|---|---|
| Read-Only 1000 | 5 min | ✅ |
| Random_points 1000 | 5 min | ✅ |
| Random_ranges 1000 | 5 min | ✅ |
| Read-Write 1000 | 5 min | ✅ |
| Write_only 1000 | 5 min | ✅ |
| Update-index 1000 | 5 min | ✅ |
| Update-Non-index 1000 | 5 min | ✅ |
| Delete 1000 | 5 min | ✅ |
| Insert 1000 | 13 min | ✅ |
| Insert Ignore 1000 | 7 min | ✅ |
| **Insert Duplicate 1000** | **7 min** | **❌ Failed** |
## Root Cause Analysis
### Error Chain
```
CN memory spikes to 40-41 GiB (approaching 55 GiB limit)
→ DN detects memory pressure, triggers MergeExecutorEvent-PauseDueToOOMAlert (available-mem: 0)
→ CN service fails to allocate memory for new connections
→ CN reports "cannot get salt, i/o timeout" errors
→ Proxy detects closed connections: "close conneciton failed"
→ Sysbench client cannot establish new connections
→ Test times out after 7 minutes
→ Step 34 marked as failure
```
### Why Insert Duplicate Specifically?
The "Insert Duplicate" test performs `INSERT ... ON DUPLICATE KEY UPDATE` operations with 1000 concurrent threads. This workload:
1. Generates high memory pressure due to duplicate key checking and update operations
2. Causes CN memory to spike to 40-41 GiB
3. Triggers DN OOM protection (available-mem: 0)
4. Results in cascading connection failures across the cluster
Other tests (Read-Only, Random_points, etc.) completed successfully because they generate less memory pressure.
## Impact
- **Test Reliability**: Insert Duplicate test fails intermittently under high memory pressure
- **Cluster Stability**: Memory pressure causes cascading failures affecting all connections
- **User Impact**: Similar memory pressure issues could affect production workloads with high-concurrency INSERT operations
## Related Issues
This is similar to previous sysbench1000w failures caused by memory pressure:
- Insert Ignore 1000 threads also took 7 minutes (longer than normal 5 min)
- Insert 1000 threads took 13 minutes (longer than normal 5 min)
- These tests show signs of memory pressure even when they don't fail completely
## Suggested Investigation
1. **Memory Profiling**: Profile CN memory usage during Insert Duplicate workload to identify memory allocation hotspots
2. **DN OOM Threshold**: Investigate why DN triggers OOM alerts at 38.9 GiB (well below the 55 GiB limit)
3. **Connection Pool**: Check if CN connection pool properly handles memory pressure scenarios
4. **Memory Limits**: Review if 55 GiB is sufficient for high-concurrency INSERT workloads with 1000 threads
## Logs
- **CI Job**: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/29940401960/job/89124090085
- **Namespace**: `mo-branch-commit-273400583-20260723`
- **Failure Window**: 2026-07-23 06:24:24 - 06:31:37 UTC
Contributor guide
Assessment
This issue has not been assessed yet.