cockroachdb / cockroachdb/cockroach
rpc: stale NodeID=0 peer entries persist indefinitely after node restart
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Problem
During node startup, the gossip bootstrap sequence creates unvalidated DRPC/gRPC connections (`NodeID=0`) via `GRPCUnvalidatedDial` / `DRPCUnvalidatedDial`. These connections fail their initial heartbeat during the startup window (before remote peers are ready), causing them to enter the `inactive` state with a 24-hour deletion timer. Once validated connections are subsequently established, these ghost entries persist alongside the healthy connections.
Two issues compound:
1. **Ghost entries are recreated on every restart.** The gossip bootstrap sequence always dials peers without a NodeID first. During the startup window, the initial heartbeat fails before any validated sibling connection exists, so `shouldDeleteAfter` sets `deleteAfter = 24h` and the probe exits. The validated connection arrives moments later as a separate peer entry and never cleans up the ghost.
2. **The 24h reaper may never fire in a stable cluster.** Deletion of inactive entries requires `touchOldPeers()` to be called, which only happens inside `onHeartbeatFailed()` of *another* peer. If all validated connections remain healthy (no heartbeat failures), no peer ever invokes `touchOldPeers`, and the ghost entries sit past their 24h deadline indefinitely.
## Observed Behavior (drt-chaos-aws)
After enabling DRPC and performing a full cluster restart:
| Node | Healthy Conns | Inactive (ghost) | Healthy % |
|------|--------------|-------------------|-----------|
| n0001 | 22 | 6 | 78.6% |
| n0002 | 22 | 4 | 84.6% |
| n0003 | 21 | 1 | 95.5% |
| n0004 | 20 | 0 | 100% |
| n0005 | 19 | 1 | 95.0% |
| n0006 | 20 | 1 | 95.2% |
- All ghost entries have `remote_node_id=0` (or stale validated IDs) and are in `system` connection class.
- Each ghost entry has a corresponding healthy connection to the same remote address with the correct NodeID.
- Zero connections are in the `unhealthy` state.
- Values persisted unchanged for 8+ hours and were recreated identically after a full cluster restart.
- Ghost entries do not participate in heartbeats — heartbeat counts match the healthy connection count exactly.
## Root Cause
`shouldDeleteAfter()` in `pkg/rpc/peer.go`:
```go
} else if myKey.NodeID == 0 {
if !ok || !sibHealthy {
deleteAfter = 24 * time.Hour
}
// When ok && sibHealthy: deleteAfter stays 0 (Scenario B - never deleted)
// But during startup, sibling doesn't exist yet, so we hit the 24h path (Scenario A)
}
```
Combined with the reaper dependency:
- `touchOldPeers()` (the only mechanism that checks `deletable()` and triggers cleanup) is only called from `onHeartbeatFailed()`.
- Inactive peers have their probes stopped (on-demand only), so they can't trigger their own cleanup.
- In a stable cluster with no heartbeat failures, the reaper is never invoked.
## Impact
Low severity — inactive entries consume no goroutines or CPU (probes are stopped). The impact is:
- Misleading `rpc.connection.healthy` percentage in metrics/dashboards
- `rpc.connection.inactive` gauge is non-zero, which can cause confusion during incident triage
- Minor memory overhead for the dormant peer map entries
## Not DRPC-specific
This affects both gRPC and DRPC equally. The entire `peerMap`/`peer` lifecycle is generic (`peer[Conn rpcConn]`). The same ghost entries would have existed with gRPC; DRPC enablement simply drew attention to the connection metrics.
## Environment
- CockroachDB: master (current)
- Cluster: drt-chaos-aws (6 nodes)
- Affected code: `pkg/rpc/peer.go`, `pkg/rpc/peer_map.go`
Epic: none
Jira issue: CRDB-62929
Contributor guide
Assessment
This issue has not been assessed yet.