ChainSafe / ChainSafe/lodestar

Beacon node hangs on graceful shutdown, requires SIGKILL (v1.45.0)

Open
#9,744 7 comments 0 reactions 1 assignee Claimed by @spiral-ladder View on GitHub
prio-high
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 18h
Merged PRs (30d)
156

Description

### Describe the bug

When the beacon node receives a shutdown signal (`SIGTERM`/`SIGINT`, e.g. `systemctl stop`), Lodestar logs `Stopping gracefully, use Ctrl+C again to force process exit` and acknowledges the signal, but then **does not actually exit**. The process lingers for several minutes: the slot clock keeps advancing while the head stays frozen (`head: (slot -1)`, `(slot -2)`, … `(slot -33)`), peers drop off, and it eventually falls into `Searching peers`. The only way to stop it is to escalate all the way to `SIGKILL`.

Reported by two independent mainnet operators, both confirming it on **v1.45.0**. It also reproduced on the prior release, so this is not a v1.45.0 regression — but it is still present in v1.45.0.

Surfaced downstream in EthPillar: https://github.com/mjkeating/EthPillar/issues/46 (see https://github.com/mjkeating/EthPillar/issues/46#issuecomment-5153695991).

### To reproduce

1. Run a synced mainnet Lodestar beacon node (reported with Besu EL).
2. Send a shutdown signal (`systemctl stop consensus`, or `SIGTERM`/`SIGINT`).
3. The node logs `Stopping gracefully…` and acknowledges the signal, but the process never exits and keeps running (head frozen, slots advancing, then `Searching peers`).
4. It has to be force-killed (`systemctl kill -s SIGKILL`).

### Logs

Reporter 1 (stopping to update, ~10 min from first stop request to SIGKILL):

```
Aug 01 12:12:36 systemd[1]: Stopping consensus.service ...
Aug 01 12:12:36 lodestar[1413]: Stopping gracefully, use Ctrl+C again to force process exit
Aug 01 12:12:53 lodestar[1413]: Synced - slot: 14896262 - head: (slot -1) 0xb090…6819 ...
...
Aug 01 12:19:56 systemd[1]: Sent signal SIGINT to main process 1413 on client request.
Aug 01 12:19:56 lodestar[1413]: Forcing process exit
Aug 01 12:22:03 systemd[1]: Sent signal SIGTERM to main process 1413 on client request.
Aug 01 12:22:59 systemd[1]: Sent signal SIGKILL to main process 1413 on client request.
Aug 01 12:23:00 systemd[1]: consensus.service: Main process exited, code=killed, status=9/KILL
Aug 01 12:23:00 systemd[1]: consensus.service: Consumed ... 14.4G memory peak, 2.7G memory swap peak.
```

Reporter 2 (explicitly confirmed on v1.45.0, still alive & "searching peers" ~6.5 min after stop):

```
Aug 01 22:08:12 systemd[1]: Stopping consensus.service ...
Aug 01 22:08:12 lodestar[453]: Stopping gracefully, use Ctrl+C again to force process exit
Aug 01 22:08:29 lodestar[453]: Synced - slot: 14899840 - head: (slot -1) 0xb8fb…5a82 ... peers: 39
...
Aug 01 22:14:41 lodestar[453]: Synced - slot: 14899871 - head: (slot -32) 0xb8fb…5a82 ... peers: 39
Aug 01 22:14:53 lodestar[453]: Searching peers - peers: 39 - slot: 14899872 - head: (slot -33) ...
```

Note the head hash stays pinned (`0xb8fb…5a82`) while the slot advances — block import has stopped, but the process itself refuses to exit.

### Expected behavior

After receiving `SIGTERM`/`SIGINT`, the node should close cleanly and the process should exit within a bounded time.

### Environment

- **Lodestar:** v1.45.0 (confirmed by two reporters); also seen on the prior release
- **Network:** Mainnet
- **EL:** Besu (reporter 1)
- **OS:** Debian 13 (reporter 1)

### Possible area (hypothesis, not confirmed)

The shutdown path is:

`onGracefulShutdown()` (`packages/cli/src/util/process.ts`) → cleanup callback → `abortController.abort()` → abort listener → `await node.close()` → `process.exit(0)` (`packages/cli/src/cmds/beacon/handler.ts`).

The explicit `process.exit(0)` only runs **after** `node.close()` resolves. If `node.close()` hangs on a subsystem that never settles, the process never exits — which matches the observed behavior. This looks related to the historical "active handles / worker not shutting down" problems (#5642, #5775, #7378).

Two things worth considering:
- Root-cause the subsystem in `node.close()` that fails to settle.
- As a safety net, add a bounded close timeout that forces `process.exit` if `node.close()` doesn't resolve within N seconds, so shutdown is always deterministic even if a subsystem hangs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.