matrixorigin / matrixorigin/matrixone

[Bug] sysbench1000w Insert Duplicate fails: CN memory pressure (40-41 GiB) causes cascading connection failures

Open
#26,088 4 comments 0 reactions 1 assignee Claimed by @ck89119 View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Issue

sysbench1000w "Insert Duplicate 1000 threads" test fails with connection timeouts due to CN memory pressure. During the test, CN memory spikes to 40-41 GiB (close to the 55 GiB limit), triggering DN OOM alerts and cascading connection failures across the cluster.

## Environment

| Field | Value |
|---|---|
| **CI Run** | https://github.com/matrixorigin/mo-nightly-regression/actions/runs/29940401960/job/89124090085 |
| **MO Commit** | `273400583d84d8da0ea2d61d9cb1c3dbd75db708` (4.2-dev) |
| **Namespace** | `mo-branch-commit-273400583-20260723` |
| **Cluster** | TKE (3 CN + 1 DN + 2 Proxy) |
| **Failure Time** | 2026-07-23 06:24:24 - 06:31:37 UTC |
| **Failed Step** | Step 34: Start OLTP Insert Duplicate Test No Auto_increment With 1000 Threads |

## Failure Details

**Step 34** ran for 7 minutes (06:24:24 - 06:31:37 UTC) before timing out. The test normally completes in 5-7 minutes.

### Error Messages

1. **CN service error** (06:31:56 on CN `gr4jr`):
```
cannot get salt, maybe not use proxy
error: read tcp 10.143.140.190:6001->10.143.140.182:49052: i/o timeout
```

2. **Proxy connection failures** (06:31:50-55):
```
close conneciton failed
error: close tcp4 10.143.140.186:6001->10.128.1.138:42544: use of closed network connection
```

3. **DN OOM alerts** (06:31:53-59):
```
MergeExecutorEvent-PauseDueToOOMAlert
available-mem: 0
```

### Memory Usage at Failure Time

**CN Pods** (06:28-06:30 UTC peak):

| CN Pod | Peak Memory | Time |
|---|---|---|
| `7f25f` | 41.4 GiB | 06:28 |
| `gr4jr` | 40.8 GiB | 06:30 |
| `cznhr` | 40.5 GiB | 06:25 |

**DN Pod**: 38.9 GiB peak at 06:31

### Normal vs Failed Tests

| Test | Duration | Result |
|---|---|---|
| Read-Only 1000 | 5 min | ✅ |
| Random_points 1000 | 5 min | ✅ |
| Random_ranges 1000 | 5 min | ✅ |
| Read-Write 1000 | 5 min | ✅ |
| Write_only 1000 | 5 min | ✅ |
| Update-index 1000 | 5 min | ✅ |
| Update-Non-index 1000 | 5 min | ✅ |
| Delete 1000 | 5 min | ✅ |
| Insert 1000 | 13 min | ✅ |
| Insert Ignore 1000 | 7 min | ✅ |
| **Insert Duplicate 1000** | **7 min** | **❌ Failed** |

## Root Cause Analysis

### Error Chain

```
CN memory spikes to 40-41 GiB (approaching 55 GiB limit)
→ DN detects memory pressure, triggers MergeExecutorEvent-PauseDueToOOMAlert (available-mem: 0)
→ CN service fails to allocate memory for new connections
→ CN reports "cannot get salt, i/o timeout" errors
→ Proxy detects closed connections: "close conneciton failed"
→ Sysbench client cannot establish new connections
→ Test times out after 7 minutes
→ Step 34 marked as failure
```

### Why Insert Duplicate Specifically?

The "Insert Duplicate" test performs `INSERT ... ON DUPLICATE KEY UPDATE` operations with 1000 concurrent threads. This workload:
1. Generates high memory pressure due to duplicate key checking and update operations
2. Causes CN memory to spike to 40-41 GiB
3. Triggers DN OOM protection (available-mem: 0)
4. Results in cascading connection failures across the cluster

Other tests (Read-Only, Random_points, etc.) completed successfully because they generate less memory pressure.

## Impact

- **Test Reliability**: Insert Duplicate test fails intermittently under high memory pressure
- **Cluster Stability**: Memory pressure causes cascading failures affecting all connections
- **User Impact**: Similar memory pressure issues could affect production workloads with high-concurrency INSERT operations

## Related Issues

This is similar to previous sysbench1000w failures caused by memory pressure:
- Insert Ignore 1000 threads also took 7 minutes (longer than normal 5 min)
- Insert 1000 threads took 13 minutes (longer than normal 5 min)
- These tests show signs of memory pressure even when they don't fail completely

## Suggested Investigation

1. **Memory Profiling**: Profile CN memory usage during Insert Duplicate workload to identify memory allocation hotspots
2. **DN OOM Threshold**: Investigate why DN triggers OOM alerts at 38.9 GiB (well below the 55 GiB limit)
3. **Connection Pool**: Check if CN connection pool properly handles memory pressure scenarios
4. **Memory Limits**: Review if 55 GiB is sufficient for high-concurrency INSERT workloads with 1000 threads

## Logs

- **CI Job**: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/29940401960/job/89124090085
- **Namespace**: `mo-branch-commit-273400583-20260723`
- **Failure Window**: 2026-07-23 06:24:24 - 06:31:37 UTC

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.