ChainSafe / ChainSafe/lodestar
Beacon node hangs on graceful shutdown, requires SIGKILL (v1.45.0)
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 156
Description
### Describe the bug
When the beacon node receives a shutdown signal (`SIGTERM`/`SIGINT`, e.g. `systemctl stop`), Lodestar logs `Stopping gracefully, use Ctrl+C again to force process exit` and acknowledges the signal, but then **does not actually exit**. The process lingers for several minutes: the slot clock keeps advancing while the head stays frozen (`head: (slot -1)`, `(slot -2)`, … `(slot -33)`), peers drop off, and it eventually falls into `Searching peers`. The only way to stop it is to escalate all the way to `SIGKILL`.
Reported by two independent mainnet operators, both confirming it on **v1.45.0**. It also reproduced on the prior release, so this is not a v1.45.0 regression — but it is still present in v1.45.0.
Surfaced downstream in EthPillar: https://github.com/mjkeating/EthPillar/issues/46 (see https://github.com/mjkeating/EthPillar/issues/46#issuecomment-5153695991).
### To reproduce
1. Run a synced mainnet Lodestar beacon node (reported with Besu EL).
2. Send a shutdown signal (`systemctl stop consensus`, or `SIGTERM`/`SIGINT`).
3. The node logs `Stopping gracefully…` and acknowledges the signal, but the process never exits and keeps running (head frozen, slots advancing, then `Searching peers`).
4. It has to be force-killed (`systemctl kill -s SIGKILL`).
### Logs
Reporter 1 (stopping to update, ~10 min from first stop request to SIGKILL):
```
Aug 01 12:12:36 systemd[1]: Stopping consensus.service ...
Aug 01 12:12:36 lodestar[1413]: Stopping gracefully, use Ctrl+C again to force process exit
Aug 01 12:12:53 lodestar[1413]: Synced - slot: 14896262 - head: (slot -1) 0xb090…6819 ...
...
Aug 01 12:19:56 systemd[1]: Sent signal SIGINT to main process 1413 on client request.
Aug 01 12:19:56 lodestar[1413]: Forcing process exit
Aug 01 12:22:03 systemd[1]: Sent signal SIGTERM to main process 1413 on client request.
Aug 01 12:22:59 systemd[1]: Sent signal SIGKILL to main process 1413 on client request.
Aug 01 12:23:00 systemd[1]: consensus.service: Main process exited, code=killed, status=9/KILL
Aug 01 12:23:00 systemd[1]: consensus.service: Consumed ... 14.4G memory peak, 2.7G memory swap peak.
```
Reporter 2 (explicitly confirmed on v1.45.0, still alive & "searching peers" ~6.5 min after stop):
```
Aug 01 22:08:12 systemd[1]: Stopping consensus.service ...
Aug 01 22:08:12 lodestar[453]: Stopping gracefully, use Ctrl+C again to force process exit
Aug 01 22:08:29 lodestar[453]: Synced - slot: 14899840 - head: (slot -1) 0xb8fb…5a82 ... peers: 39
...
Aug 01 22:14:41 lodestar[453]: Synced - slot: 14899871 - head: (slot -32) 0xb8fb…5a82 ... peers: 39
Aug 01 22:14:53 lodestar[453]: Searching peers - peers: 39 - slot: 14899872 - head: (slot -33) ...
```
Note the head hash stays pinned (`0xb8fb…5a82`) while the slot advances — block import has stopped, but the process itself refuses to exit.
### Expected behavior
After receiving `SIGTERM`/`SIGINT`, the node should close cleanly and the process should exit within a bounded time.
### Environment
- **Lodestar:** v1.45.0 (confirmed by two reporters); also seen on the prior release
- **Network:** Mainnet
- **EL:** Besu (reporter 1)
- **OS:** Debian 13 (reporter 1)
### Possible area (hypothesis, not confirmed)
The shutdown path is:
`onGracefulShutdown()` (`packages/cli/src/util/process.ts`) → cleanup callback → `abortController.abort()` → abort listener → `await node.close()` → `process.exit(0)` (`packages/cli/src/cmds/beacon/handler.ts`).
The explicit `process.exit(0)` only runs **after** `node.close()` resolves. If `node.close()` hangs on a subsystem that never settles, the process never exits — which matches the observed behavior. This looks related to the historical "active handles / worker not shutting down" problems (#5642, #5775, #7378).
Two things worth considering:
- Root-cause the subsystem in `node.close()` that fails to settle.
- As a safety net, add a bounded close timeout that forces `process.exit` if `node.close()` doesn't resolve within N seconds, so shutdown is always deterministic even if a subsystem hangs.
Contributor guide
Assessment
This issue has not been assessed yet.