cockroachdb / cockroachdb/cockroach

rpc: stale NodeID=0 peer entries persist indefinitely after node restart

Open
#168,381 0 comments 0 reactions 0 assignees View on GitHub
branch-master C-bug T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Problem

During node startup, the gossip bootstrap sequence creates unvalidated DRPC/gRPC connections (`NodeID=0`) via `GRPCUnvalidatedDial` / `DRPCUnvalidatedDial`. These connections fail their initial heartbeat during the startup window (before remote peers are ready), causing them to enter the `inactive` state with a 24-hour deletion timer. Once validated connections are subsequently established, these ghost entries persist alongside the healthy connections.

Two issues compound:

1. **Ghost entries are recreated on every restart.** The gossip bootstrap sequence always dials peers without a NodeID first. During the startup window, the initial heartbeat fails before any validated sibling connection exists, so `shouldDeleteAfter` sets `deleteAfter = 24h` and the probe exits. The validated connection arrives moments later as a separate peer entry and never cleans up the ghost.

2. **The 24h reaper may never fire in a stable cluster.** Deletion of inactive entries requires `touchOldPeers()` to be called, which only happens inside `onHeartbeatFailed()` of *another* peer. If all validated connections remain healthy (no heartbeat failures), no peer ever invokes `touchOldPeers`, and the ghost entries sit past their 24h deadline indefinitely.

## Observed Behavior (drt-chaos-aws)

After enabling DRPC and performing a full cluster restart:

| Node | Healthy Conns | Inactive (ghost) | Healthy % |
|------|--------------|-------------------|-----------|
| n0001 | 22 | 6 | 78.6% |
| n0002 | 22 | 4 | 84.6% |
| n0003 | 21 | 1 | 95.5% |
| n0004 | 20 | 0 | 100% |
| n0005 | 19 | 1 | 95.0% |
| n0006 | 20 | 1 | 95.2% |

- All ghost entries have `remote_node_id=0` (or stale validated IDs) and are in `system` connection class.
- Each ghost entry has a corresponding healthy connection to the same remote address with the correct NodeID.
- Zero connections are in the `unhealthy` state.
- Values persisted unchanged for 8+ hours and were recreated identically after a full cluster restart.
- Ghost entries do not participate in heartbeats — heartbeat counts match the healthy connection count exactly.

## Root Cause

`shouldDeleteAfter()` in `pkg/rpc/peer.go`:

```go
} else if myKey.NodeID == 0 {
if !ok || !sibHealthy {
deleteAfter = 24 * time.Hour
}
// When ok && sibHealthy: deleteAfter stays 0 (Scenario B - never deleted)
// But during startup, sibling doesn't exist yet, so we hit the 24h path (Scenario A)
}
```

Combined with the reaper dependency:

- `touchOldPeers()` (the only mechanism that checks `deletable()` and triggers cleanup) is only called from `onHeartbeatFailed()`.
- Inactive peers have their probes stopped (on-demand only), so they can't trigger their own cleanup.
- In a stable cluster with no heartbeat failures, the reaper is never invoked.

## Impact

Low severity — inactive entries consume no goroutines or CPU (probes are stopped). The impact is:
- Misleading `rpc.connection.healthy` percentage in metrics/dashboards
- `rpc.connection.inactive` gauge is non-zero, which can cause confusion during incident triage
- Minor memory overhead for the dormant peer map entries

## Not DRPC-specific

This affects both gRPC and DRPC equally. The entire `peerMap`/`peer` lifecycle is generic (`peer[Conn rpcConn]`). The same ghost entries would have existed with gRPC; DRPC enablement simply drew attention to the connection metrics.

## Environment

- CockroachDB: master (current)
- Cluster: drt-chaos-aws (6 nodes)
- Affected code: `pkg/rpc/peer.go`, `pkg/rpc/peer_map.go`

Epic: none

Jira issue: CRDB-62929

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.