HarperFast / HarperFast/harper-pro

Self-restart can deadlock, leaving the process alive with no listeners and no container restart

Open
#696 1 comment 0 reactions 1 assignee Claimed by @DavidCockerill View on GitHub
bug
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

Harper restarts itself by exiting the process and relying on the container runtime to bring it back. When that exit deadlocks, nothing restarts it: the process stays alive with its listeners torn down, the container runtime sees no exit so it never restarts, and `docker ps` reports the container as healthy. The instance is dark indefinitely with no error surfaced anywhere.

Observed twice within 48 hours on the same instance (5.1.19), each time following a component deployment:

```
12:41:59 Component '' ... failed to replicate to 1 of 1 peer node(s) (500, phase 'restart')
12:41:59 Restarting Harper. This may take up to 60 seconds.
12:42:01 Replacement worker exited before starting; leaving the existing worker in place -1
12:42:01 Exiting Harper process to trigger a container restart
<- log ends here; process still alive 31 hours later
```

The replacement worker exited **58ms** after the restart began — too fast to have loaded the component — and no error from the component appears anywhere in the logs.

## Evidence of the deadlock

`/proc//task/*/wchan` at the time (18 threads):

```
MainThread S futex_wait_queue <- blocked; no JS can run
DelayedTaskSche S ep_poll
V8Worker x4 S futex_wait_queue
SignalInspector S futex_wait_queue
libuv-worker x4 S futex_wait_queue
rocksdb:low x4 S futex_wait_queue
rocksdb:high S futex_wait_queue
InspectorIo S ep_poll
http S futex_wait_queue
```

MainThread kernel stack: `futex_wait_queue → __futex_wait → futex_wait → do_futex → __x64_sys_futex`. Sixteen of eighteen threads parked on futexes; only V8's delayed-task scheduler and the inspector still cycling, neither of which runs JS.

## How to recognise it

Everything external looks correct, which is what makes this expensive to diagnose:

- container reports running, `RestartCount=0`, process alive for weeks
- inside the container's network namespace, **only the debug inspector port is bound** — every service port (HTTPS, operations, replication, MQTT) is gone
- probing the instance's own port gives a **fast** TLS error (`curl` exit 35, ~3ms), because the host-side port mapping still accepts TCP with nothing behind it. This is easily misread as a certificate problem.
- the operations socket gives connection-refused
- logs stop dead at the `Exiting Harper process` line

## Impact beyond the instance

A dark node stops accepting replication, so its peer's write transactions queue without bound (`Outstanding write transactions have too long of queue`, 503s to callers). That queue is held in memory; on a memory-limited deployment the peer eventually crosses its cgroup `memory.high` and the kernel begins parking its threads rather than OOM-killing it. The peer then stops serving as well, and is too stalled to recover on its own.

In the observed case one wedged instance took down both nodes of a two-node cluster for 31 hours, and its memory pressure degraded unrelated tenants sharing the host.

## Proposed fix

The self-restart path has no backstop when the exit does not happen — no watchdog, no forced exit, no signal that the listeners are gone.

A watchdog armed at the start of shutdown that force-exits if the process is still alive after N seconds would convert a permanent silent outage into a container restart. **It cannot be a `setTimeout`**: the main thread is futex-blocked and the event loop is not cycling, so a JS timer never fires. It has to run somewhere still scheduled — a dedicated watchdog thread issuing `process.kill(process.pid, 'SIGKILL')`, or equivalent.

Separately worth investigating: why `Replacement worker exited before starting` occurs at all, and what lock the shutdown blocks on. That has a reliable reproduction — deploying a component while the peer is unreachable — but is likely a deeper fix than the backstop.

## Workaround

`docker restart` the affected container. The peer will **not** recover on its own once it has begun memory-parking; it needs its own restart afterwards.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.