matrixorigin / matrixorigin/matrixone
CN heartbeat failure during HAKeeper isolation: no self-termination leads to zombie CN
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
## Summary
When a CN node loses connectivity to HAKeeper (while CN↔TN and CN↔Proxy remain healthy), the heartbeat task logs the error but does **not** trigger process shutdown or mark the CN as unhealthy. The CN process continues running indefinitely — holding all in-flight transactions, locks, and offheap workspace allocations — while HAKeeper marks it as expired and removes it from the routing table.
## Root Cause
`cnservice/server_heartbeat.go:heartbeat()` (line 102-106):
```go
func (s *service) heartbeat(ctx context.Context) {
hb := s.collectHeartbeat(ctx)
ctx2, cancel := context.WithTimeout(ctx, cnHeartbeatTimeout)
defer cancel()
if _, err := s.managedHAKeeperClient.SendCNHeartbeat(ctx2, hb); err != nil {
s.logger.Error("failed to send cn heartbeat", zap.Error(err))
return // ← only logs, no stopper.Stop() or health status change
}
}
```
HAKeeper expires the CN after 30s (deleteCNWait) and removes it from ClusterDetails. Proxy stops routing new connections. But the CN process itself never knows it has been evicted.
## Impact
| Resource | State |
|----------|-------|
| In-flight transactions | Remain in memory; sender retries TN RPCs (30s budget) |
| LockService locks | Remote locks held for up to 10min (orphan lock timeout) |
| Workspace / offheap memory | Never freed (no delTransaction, no process exit) |
| txnClient goroutines | All alive, including pausers/waiters |
## Trigger Condition
- CN ↔ HAKeeper network partition lasting > 30s
- CN ↔ TN and CN ↔ Proxy connectivity remain healthy
## Related Issues
- [#25198](https://github.com/matrixorigin/matrixone/issues/25198) — FinalizeCommitWithUnknownResult workspace leak
- [#25195](https://github.com/matrixorigin/matrixone/issues/25195) — gcZombieTxn cleanup chain
## Suggested Fix
1. After N consecutive heartbeat failures, call `stopper.Stop()` to initiate graceful CN shutdown
2. Or: update a health status that causes the CN to reject new work and drain existing transactions
## Severity: MEDIUM
HAKeeper-side eviction protects client correctness (new connections routed away). The risk is silent resource waste — the zombie CN holds memory/quota until process restart.
Contributor guide
Assessment
This issue has not been assessed yet.