HarperFast / HarperFast/harper-pro
SIGSEGV in rocksdb-js finalizer during worker-thread teardown when a replication peer is stopped
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
A Node worker thread shutting down segfaulted inside the RocksDB N-API addon's finalizer, killing PID 1 and taking the container down. It happened when the **replication peer was stopped** — i.e. during a step that a rolling upgrade performs on purpose, on every node.
harper-pro 5.2.3, `@harperfast/rocksdb-js` **2.7.1**, RocksDB storage (post LMDB migration), two-node replicating pair, Linux x64 glibc.
## Stack
```
PID 1 received SIGSEGV for address: 0x2d9000000e1
segfault-handler.node(+0x2e87)
libc.so.6(+0x3fdf0)
node(napi_get_reference_value+0x35)
@harperfast/rocksdb-js-linux-x64-glibc/rocksdb-js.node(+0x398b98)
node(node_napi_env__::CallFinalizer(void (*)(napi_env__*, void*, void*), void*, void*)+0x6a)
node(+0x8c2f34)
node(node::Environment::RunCleanup()+0x9f)
node(node::FreeEnvironment(node::Environment*)+0x79)
node(node::worker::Worker::Run()+0xd0c)
node(+0xaa1e76)
libc.so.6(+0x92b7b)
libc.so.6(__clone+0x40)
```
Read bottom-up: a worker thread is exiting → `FreeEnvironment` → `RunCleanup` → napi finalizers → the addon calls `napi_get_reference_value` on a reference whose environment is already being torn down → fault on a bogus address. No JavaScript frames are involved.
## Trigger — correlates with the peer going away
Log immediately before, hostnames genericised:
```
16:39:44 [warn] Reconciling replication subscriptions for nodes wedged disconnected on a live worker: ['wss://node-b:9933']
16:39:44 [warn] Reconciling 3 wedged subscription(s) for wss://node-b:9933 (staggered over 150ms)
16:39:45 [error][replication] Error in connection to wss://node-b:9933 due to WebSocket was closed before the connection was established (x3, one per http worker)
16:40:06 PID 1 received SIGSEGV
```
Node B (the replication peer) had just been stopped for its own upgrade. Worker threads were being recycled as part of the wedged-subscription reconciliation, and one segfaulted on the way out. The process died; the container had to be started again (`RestartCount=0`, so no restart policy fired).
## Why this matters operationally
Stopping one node of a replicating pair is a *required step* of a rolling version upgrade. If this reproduces, upgrading node B can take down node A — the opposite of what a rolling upgrade is for. It fired once in one attempt here, so treat it as plausible-but-unconfirmed rather than deterministic.
## Version signal (suggestive, not conclusive)
| node | harper-pro | rocksdb-js | uptime on RocksDB | SIGSEGV |
| --- | --- | --- | --- | --- |
| single-node dev | 5.2.0 | **2.6.1** | 8 days | 0 |
| node A of a pair | 5.2.3 | **2.7.1** | minutes | 1 |
Two variables differ, so this is not attribution: the dev node is standalone and therefore never experiences a replication peer disconnecting, which is the observed trigger. Noting it because `harper-pro@main` pins `@harperfast/rocksdb-js: 2.7.1`, so this is the version production upgrades will land on.
## Questions
1. Is a finalizer-time `napi_get_reference_value` safe during `Environment::RunCleanup()`? If the addon keeps a `napi_ref` that can outlive env teardown, the reference should presumably be cleared on env-teardown notification (`napi_add_env_cleanup_hook`) rather than dereferenced from the finalizer.
2. Is anything known to have changed here between rocksdb-js 2.6.1 and 2.7.1?
3. Is worker recycling during replication-subscription reconciliation expected? If workers are torn down whenever a peer wedges, this finalizer path will be exercised routinely on any multi-node deployment, not just during upgrades.
## Environment notes
RocksDB came back cleanly on restart and the node has been stable since, so no data loss is apparent (WAL recovery presumably handled it) — but it was a hard crash, so worth confirming that is guaranteed rather than lucky.
Contributor guide
Assessment
This issue has not been assessed yet.