matrixorigin / matrixorigin/matrixone

[Network Partition] CN→TN RPC retry thundering herd during network flapping

Open
#25,209 0 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
deferred kind/bug
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary

During network flapping (intermittent CN↔TN partitions), every active transaction's sender independently enters its retry loop with exponential backoff (300ms → 600ms → ... → 30s budget). When the network recovers, all goroutines simultaneously attempt to create new morpc backend connections, causing a thundering herd on the TN side.

## Root Cause

`pkg/txn/rpc/sender.go` — each txn's `doSend()` retry loop operates independently:

```go
defaultMaxWaitTimeOnRetryBackendSend = 30 * time.Second // per-request budget
defaultRetryBackoff = 300 * time.Millisecond

func (s *sender) doSend(ctx context.Context, ...) {
for {
...
f, err := s.createStream(ctx)
if err != nil {
s.waitToRetrySend(ctx, ...) // independent backoff per sender
continue
}
...
s.waitToRetrySend(ctx, ...)
}
}
```

- Each txn has its own retry timer and backoff state
- No global rate limiter or circuit breaker on CN→TN RPC path
- `morpc.MaxBackendPerHost` limits connections per CN→TN, but the retry loop spins before backend creation

## Impact

Flapping cycle (~5s period):
```
t=0: Partition → 100 active txns enter retry simultaneously
t=1: 100 goroutines spinning with staggered backoff
t=3: Network recovers → thundering herd of connection re-establishment
t=5: Re-partition → cycle repeats
```

- CPU waste from idle retry spinning
- Connection storm on TN during recovery windows
- Log noise from repeated RPC failures

## Mitigating Factors

- Bounded by `MaxActiveTxn` (default 100)
- `morpc.MaxBackendPerHost` limits established connections
- Exponential backoff with jitter provides natural fan-out

## Suggested Fix

1. Add global backoff/rate limiter shared across all senders on the same CN→TN target
2. Or: circuit breaker pattern — after N consecutive failures to a specific TN, pause all senders to that TN for a cooldown period

## Severity: MEDIUM

Bounded by MaxActiveTxn but potentially degrades performance during prolonged network instability.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.