HarperFast / HarperFast/harper-pro

SIGSEGV in rocksdb-js finalizer during worker-thread teardown when a replication peer is stopped

Open
#730 2 comments 0 reactions 1 assignee Claimed by @cb1kenobi View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

A Node worker thread shutting down segfaulted inside the RocksDB N-API addon's finalizer, killing PID 1 and taking the container down. It happened when the **replication peer was stopped** — i.e. during a step that a rolling upgrade performs on purpose, on every node.

harper-pro 5.2.3, `@harperfast/rocksdb-js` **2.7.1**, RocksDB storage (post LMDB migration), two-node replicating pair, Linux x64 glibc.

## Stack

```
PID 1 received SIGSEGV for address: 0x2d9000000e1
segfault-handler.node(+0x2e87)
libc.so.6(+0x3fdf0)
node(napi_get_reference_value+0x35)
@harperfast/rocksdb-js-linux-x64-glibc/rocksdb-js.node(+0x398b98)
node(node_napi_env__::CallFinalizer(void (*)(napi_env__*, void*, void*), void*, void*)+0x6a)
node(+0x8c2f34)
node(node::Environment::RunCleanup()+0x9f)
node(node::FreeEnvironment(node::Environment*)+0x79)
node(node::worker::Worker::Run()+0xd0c)
node(+0xaa1e76)
libc.so.6(+0x92b7b)
libc.so.6(__clone+0x40)
```

Read bottom-up: a worker thread is exiting → `FreeEnvironment` → `RunCleanup` → napi finalizers → the addon calls `napi_get_reference_value` on a reference whose environment is already being torn down → fault on a bogus address. No JavaScript frames are involved.

## Trigger — correlates with the peer going away

Log immediately before, hostnames genericised:

```
16:39:44 [warn] Reconciling replication subscriptions for nodes wedged disconnected on a live worker: ['wss://node-b:9933']
16:39:44 [warn] Reconciling 3 wedged subscription(s) for wss://node-b:9933 (staggered over 150ms)
16:39:45 [error][replication] Error in connection to wss://node-b:9933 due to WebSocket was closed before the connection was established (x3, one per http worker)
16:40:06 PID 1 received SIGSEGV
```

Node B (the replication peer) had just been stopped for its own upgrade. Worker threads were being recycled as part of the wedged-subscription reconciliation, and one segfaulted on the way out. The process died; the container had to be started again (`RestartCount=0`, so no restart policy fired).

## Why this matters operationally

Stopping one node of a replicating pair is a *required step* of a rolling version upgrade. If this reproduces, upgrading node B can take down node A — the opposite of what a rolling upgrade is for. It fired once in one attempt here, so treat it as plausible-but-unconfirmed rather than deterministic.

## Version signal (suggestive, not conclusive)

| node | harper-pro | rocksdb-js | uptime on RocksDB | SIGSEGV |
| --- | --- | --- | --- | --- |
| single-node dev | 5.2.0 | **2.6.1** | 8 days | 0 |
| node A of a pair | 5.2.3 | **2.7.1** | minutes | 1 |

Two variables differ, so this is not attribution: the dev node is standalone and therefore never experiences a replication peer disconnecting, which is the observed trigger. Noting it because `harper-pro@main` pins `@harperfast/rocksdb-js: 2.7.1`, so this is the version production upgrades will land on.

## Questions

1. Is a finalizer-time `napi_get_reference_value` safe during `Environment::RunCleanup()`? If the addon keeps a `napi_ref` that can outlive env teardown, the reference should presumably be cleared on env-teardown notification (`napi_add_env_cleanup_hook`) rather than dereferenced from the finalizer.
2. Is anything known to have changed here between rocksdb-js 2.6.1 and 2.7.1?
3. Is worker recycling during replication-subscription reconciliation expected? If workers are torn down whenever a peer wedges, this finalizer path will be exercised routinely on any multi-node deployment, not just during upgrades.

## Environment notes

RocksDB came back cleanly on restart and the node has been stable since, so no data loss is apparent (WAL recovery presumably handled it) — but it was a hard crash, so worth confirming that is guaranteed rather than lucky.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.