matrixorigin / matrixorigin/matrixone
[Network Partition] CN→TN RPC retry thundering herd during network flapping
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
During network flapping (intermittent CN↔TN partitions), every active transaction's sender independently enters its retry loop with exponential backoff (300ms → 600ms → ... → 30s budget). When the network recovers, all goroutines simultaneously attempt to create new morpc backend connections, causing a thundering herd on the TN side.
## Root Cause
`pkg/txn/rpc/sender.go` — each txn's `doSend()` retry loop operates independently:
```go
defaultMaxWaitTimeOnRetryBackendSend = 30 * time.Second // per-request budget
defaultRetryBackoff = 300 * time.Millisecond
func (s *sender) doSend(ctx context.Context, ...) {
for {
...
f, err := s.createStream(ctx)
if err != nil {
s.waitToRetrySend(ctx, ...) // independent backoff per sender
continue
}
...
s.waitToRetrySend(ctx, ...)
}
}
```
- Each txn has its own retry timer and backoff state
- No global rate limiter or circuit breaker on CN→TN RPC path
- `morpc.MaxBackendPerHost` limits connections per CN→TN, but the retry loop spins before backend creation
## Impact
Flapping cycle (~5s period):
```
t=0: Partition → 100 active txns enter retry simultaneously
t=1: 100 goroutines spinning with staggered backoff
t=3: Network recovers → thundering herd of connection re-establishment
t=5: Re-partition → cycle repeats
```
- CPU waste from idle retry spinning
- Connection storm on TN during recovery windows
- Log noise from repeated RPC failures
## Mitigating Factors
- Bounded by `MaxActiveTxn` (default 100)
- `morpc.MaxBackendPerHost` limits established connections
- Exponential backoff with jitter provides natural fan-out
## Suggested Fix
1. Add global backoff/rate limiter shared across all senders on the same CN→TN target
2. Or: circuit breaker pattern — after N consecutive failures to a specific TN, pause all senders to that TN for a cooldown period
## Severity: MEDIUM
Bounded by MaxActiveTxn but potentially degrades performance during prolonged network instability.
Contributor guide
Assessment
This issue has not been assessed yet.