ChainSafe / ChainSafe/lodestar

[bun] Networking errors escape as uncaught exceptions and N-API teardown can abort

Open
#9,900 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

## Describe the bug

A five-node Lodestar fleet running under Bun is healthy enough to advance and maintain peers, but it shows several runtime-specific failures in networking error handling and shutdown paths. A same-window comparison against all non-Bun beacon instances found none of the corresponding errors.

Observed on 2026-08-21 from approximately 15:35–16:18 UTC:

### 1. Expected network aborts escape as global uncaught exceptions

```text
uncaughtException: The operation was aborted
```

- 427 occurrences across all five Bun instances in a 30-minute comparison window.
- Zero occurrences across all non-Bun beacon instances in the same window.
- Representative stack:

```text
destroyWhenAborted (node:net:88:33)
@libp2p/utils queue/job.js
any-signal
uncaughtException
```

This appears related to the abort behavior previously reported in #7280, but the issue remains reproducible at fleet scale.

### 2. Bun network error shape breaks the req/resp peer-scoring path

```text
TypeError: undefined is not an object
(evaluating 'e.type.error.message.includes')
```

Representative stack:

```text
packages/reqresp/src/request/index.ts:160
onOutgoingReqRespError (.../network/reqresp/score.ts:41)
onOutgoingRequestError (.../ReqRespBeaconNode.ts:333)
```

The preceding request log renders the error as:

```text
error=error=[object], local=true, isTrusted=false
```

`onOutgoingReqRespError` assumes `e.type.error.message` exists, but that field is absent for this Bun error. This occurred 16 times across four of the five Bun instances and zero times on non-Bun instances.

### 3. Routine socket failures escape as global uncaught exceptions

```text
uncaughtException: recvmsg ECONNREFUSED
uncaughtException: recvmsg EHOSTUNREACH
```

- 13 occurrences across the Bun fleet in the 30-minute comparison window.
- Zero occurrences on non-Bun beacon instances.

These should remain handled dial/socket failures rather than reaching Lodestar's global uncaught-exception handler.

### 4. Native/N-API teardown can abort the process

During shutdown of one instance, after the network worker was closed and Lodestar logged `terminating network worker`:

```text
assertion `left == right` failed: failed to delete napi ref
thread '' ... panicked at rust/lib.rs:29:1
fatal runtime error: failed to initiate panic, error 5, aborting
```

This was observed on one Bun instance and not on non-Bun instances in the comparison window. The immediate trigger may be in Bun or a native dependency, but it makes graceful shutdown/restart unreliable from Lodestar's perspective.

### 5. Lower-confidence logger/EventEmitter signal

All five Bun instances emitted:

```text
MaxListenersExceededWarning:
Possible EventEmitter memory leak detected.
11 error listeners added to [DailyRotateFile]
```

No equivalent warning appeared on the non-Bun instances. This may be a Bun EventEmitter compatibility difference or a real listener leak exposed by Bun; it needs a focused Bun-vs-Node reproduction before assigning root cause.

## Expected behavior

- Expected aborts and routine socket errors are handled locally and do not reach `uncaughtException`.
- Req/resp scoring safely handles every runtime error shape without throwing a secondary `TypeError`.
- Closing the network worker and native modules completes without a native abort.
- Logger transports do not accumulate unexpected listeners.

## Steps to reproduce

1. Run Lodestar under Bun on a live network with normal peer churn and verbose networking logs.
2. Let the node establish a normal peer set and process req/resp traffic.
3. Observe abort and failed-dial paths for `The operation was aborted`, `recvmsg ECONNREFUSED`, and `recvmsg EHOSTUNREACH` reaching `uncaughtException`.
4. Stop/restart the node repeatedly to exercise network-worker and native/N-API teardown.
5. Compare the same workload under Node.js.

A smaller deterministic reproduction should separately cover:

- aborting a libp2p `node:net` operation;
- passing Bun's outgoing request error into `onOutgoingReqRespError`;
- repeated network-worker/native teardown;
- `DailyRotateFile` listener counts under Bun and Node.

## Additional context

- Fleet: five Bun beacon nodes (`feat1-*`), four Hoodi and one Mainnet.
- Lodestar: `v1.46.0/c746ec6`, commit [`c746ec6a07a98a92c7bdb96bb410db307934f7b8`](https://github.com/ChainSafe/lodestar/commit/c746ec6a07a98a92c7bdb96bb410db307934f7b8).
- Runtime: Bun. The exact Bun release was not included in the collected logs; Bun reported the Node compatibility version as `v26.3.0` through `process.version`/`nodejs_version_info`.
- OS: Linux.
- All five nodes remained up, advancing, and peered after the latest starts. The steady-state errors were noisy/recoverable; the teardown abort was the severe failure.
- Earlier Bun tracking issue: #7280.

Suggested priority:

1. Native/N-API teardown abort.
2. Defensive handling of the error shape in `onOutgoingReqRespError`.
3. Escaped abort and `recvmsg` exceptions.
4. Bun-vs-Node logger listener comparison.

Contributor guide

Open the contributing guide

Research direction

Start with packages/reqresp/src/request/index.ts:160, network/reqresp/score.ts:41, and ReqRespBeaconNode.ts:333, then reproduce Bun versus Node for the listed abort and request-error paths. Also isolate network-worker/N-API teardown and DailyRotateFile listener counts. Done means routine errors stay handled, scoring accepts Bun error shapes, shutdown does not abort, and listener counts remain bounded.

Written by the indexing model from the issue text.

Assessment

Tech stack
bun, node.js, typescript
Domain
backend, networking, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.