matrixorigin / matrixorigin/matrixone

CN heartbeat failure during HAKeeper isolation: no self-termination leads to zombie CN

Open
#25,207 0 comments 0 reactions 1 assignee Claimed by @jiangxinmeng1 View on GitHub
deferred kind/bug
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

## Summary

When a CN node loses connectivity to HAKeeper (while CN↔TN and CN↔Proxy remain healthy), the heartbeat task logs the error but does **not** trigger process shutdown or mark the CN as unhealthy. The CN process continues running indefinitely — holding all in-flight transactions, locks, and offheap workspace allocations — while HAKeeper marks it as expired and removes it from the routing table.

## Root Cause

`cnservice/server_heartbeat.go:heartbeat()` (line 102-106):

```go
func (s *service) heartbeat(ctx context.Context) {
hb := s.collectHeartbeat(ctx)
ctx2, cancel := context.WithTimeout(ctx, cnHeartbeatTimeout)
defer cancel()
if _, err := s.managedHAKeeperClient.SendCNHeartbeat(ctx2, hb); err != nil {
s.logger.Error("failed to send cn heartbeat", zap.Error(err))
return // ← only logs, no stopper.Stop() or health status change
}
}
```

HAKeeper expires the CN after 30s (deleteCNWait) and removes it from ClusterDetails. Proxy stops routing new connections. But the CN process itself never knows it has been evicted.

## Impact

| Resource | State |
|----------|-------|
| In-flight transactions | Remain in memory; sender retries TN RPCs (30s budget) |
| LockService locks | Remote locks held for up to 10min (orphan lock timeout) |
| Workspace / offheap memory | Never freed (no delTransaction, no process exit) |
| txnClient goroutines | All alive, including pausers/waiters |

## Trigger Condition

- CN ↔ HAKeeper network partition lasting > 30s
- CN ↔ TN and CN ↔ Proxy connectivity remain healthy

## Related Issues

- [#25198](https://github.com/matrixorigin/matrixone/issues/25198) — FinalizeCommitWithUnknownResult workspace leak
- [#25195](https://github.com/matrixorigin/matrixone/issues/25195) — gcZombieTxn cleanup chain

## Suggested Fix

1. After N consecutive heartbeat failures, call `stopper.Stop()` to initiate graceful CN shutdown
2. Or: update a health status that causes the CN to reject new work and drain existing transactions

## Severity: MEDIUM

HAKeeper-side eviction protects client correctness (new connections routed away). The risk is silent resource waste — the zombie CN holds memory/quota until process restart.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.