SIGTERM does not trigger graceful shutdown: container stop segfaults in TLS/fs completion callbacks, or SIGKILLs at the stop timeout

> **Suggested priority: P2** — every container replacement is an unclean shutdown (segfault or
> SIGKILL), which leans recovery on replay and leaves replication/blob state mid-flight. It also
> amplifies #683: each replacement is a guaranteed reconnect, and any latched link re-enters a
> whole-table base copy. The glibc heap-corruption signature deserves a look independent of
> teardown.

## Summary

`SIGTERM` never initiates a graceful shutdown. The registered handlers do only bookkeeping, and because registering *any* `SIGTERM` listener suppresses Node's default terminate-on-signal, the process simply keeps running until the container runtime SIGKILLs it at the stop-timeout. When there is heavy in-flight TLS work at that moment (replication blob sends), the process instead **segfaults within ~1s** in a libuv completion callback.

Observed on **harper-pro 5.2.1**, Node v24.19.0, during a rolling upgrade of a 4-node cluster: 2 of the 4 nodes segfaulted on teardown, 2 were SIGKILLed after the full 10s grace. Either way, **every container replacement is an unclean shutdown.**

## Evidence: the crashes are pre-restart teardown, not post-restart

The orchestrator creates the replacement container, stops the old one, then starts the replacement. Comparing `.Created` / `.State.StartedAt` against the kernel's segfault timestamps:

| Node | New ctr `.Created` | Old proc crash | New ctr `.StartedAt` | create→start |
|---|---|---|---|---|
| cd5 | 16:44:12.784 | — | 16:44:26.778 | **13.99s** |
| e9v | 16:50:59.317 | — | 16:51:12.649 | **13.33s** |
| v3t | 16:57:50.356 | 16:57:51.025 (**+0.669s**) | 16:57:55.329 | 4.97s |
| yc0 | 17:04:34.077 | 17:04:34.964 (**+0.886s**) | 17:04:38.065 | 3.99s |

The crash lands inside the old process's teardown window, less than a second after SIGTERM. The crashed nodes complete ~9s *faster* because the segfault short-circuits the grace period.

`docker inspect` shows `StopTimeout: ` and `StopSignal: ""` — i.e. SIGTERM, 10s grace, then SIGKILL. The clean nodes' ~13–14s (10s grace + create/start overhead) is what you would expect if the process never exits voluntarily and is SIGKILLed at the deadline.

On the crashing node, the application log runs at full tilt right up to **274ms** before the crash, with no shutdown line of any kind:

```
17:04:34.687Z [http/8] [warn] [replication]: Suppressed 541 additional blob send errors in the last 5s
17:04:34.690Z [http/8] [warn] [replication]: Error sending blob BlobReadError: Blob file not found for .../d64/52a

```

## The signal path does no shutdown work

Every `SIGTERM` registration in `dist/core`:

| Location | Handler |
|---|---|
| `bin/run.js:99` | `removeHdbPid()` |
| `utility/processManagement/processManagement.js:70` | `cleanupChildrenProcesses` |
| `server/threads/threadServer.js:43` | `inspector.close()` |

None calls `shutdownWorkers()`, closes servers, or drains/closes databases. The graceful-drain protocol in `manageThreads.js` (`EXTEND_SHUTDOWN_DEADLINE`, whose comment explicitly names *"in-flight work (e.g. replication blob sends)"*) is only reachable from the **in-process worker-restart** path — config reload, component deploy — never from a signal.

Because a `SIGTERM` listener exists, Node does not apply its default exit behavior, so nothing terminates the process on SIGTERM at all.

## Crash signatures — all in-flight libuv completions against a torn-down environment

14 days of kernel logs across the 4 nodes. All crashes are in threads named `http` (worker threads), inside `node::worker::Worker::Run`. **No OOM kills on any node.**

**yc0, Aug 10 17:04:34 — `segfault at 302a350 ... error 7` (user-mode write):** TLS write-completion callback into a disposed environment.
```
node::InternalMakeCallback
node::AsyncWrap::MakeCallback
node::ReportWritesToJSStreamListener::OnStreamAfterReqFinished
node::WriteWrap::OnDone
node::crypto::TLSWrap::InvokeQueued
node::crypto::TLSWrap::OnStreamAfterWrite
node::Environment::RunAndClearNativeImmediates
node::Environment::CheckImmediate → uv_run → node::worker::Worker::Run
```

**v3t, Aug 10 16:57:51 — `general protection fault ... in libc`:** glibc heap-corruption detection (`malloc_printerr` → `abort`) while OpenSSL allocated for a TLS record read.
```
abort ← libc (__libc_message / malloc_printerr / _int_malloc) ← malloc
CRYPTO_malloc ← tls_get_more_records ← tls_read_record
ssl3_read_bytes ← ssl3_read ← SSL_read
node::crypto::TLSWrap::ClearOut
node::crypto::TLSWrap::OnStreamRead
node::LibuvStreamWrap::OnUvRead → uv_run → node::worker::Worker::Run
```
Note this is the *detection* site, not the corruption site.

**cd5, Aug 6 00:13:06 — `segfault at 19 ... error 4` (user-mode read):** fs request callback destructor on a freed/garbage pointer.
```
node::fs::FSReqCallback::~FSReqCallback() [deleting dtor]
uv__work_done → uv_run → node::worker::Worker::Run
```

**cd5, Aug 1 03:44:51 — `trap int3`:** V8 CHECK/abort.

Three different subsystems (TLS write completion, TLS record read, fs request teardown), one shape: a libuv completion running after its environment/handles have been disposed.

## Why only 2 of 4 nodes

The two that crashed were the two saturating blob replication over TLS at teardown — the crashing node was logging *"Suppressed 541–737 additional blob send errors in the last 5s"* continuously up to the moment of death. More in-flight TLS writes when the axe falls ⇒ higher chance of a completion firing into a disposed environment. The two quiet nodes rode out the 10s and were SIGKILLed cleanly.

## Impact

- The process was being terminated anyway, so no direct request-level impact from the crash itself.
- But **every container replacement is an unclean shutdown** — segfault, or SIGKILL at the stop timeout. Recovery leans on audit-log replay instead of a clean close.
- Replication/blob state is left mid-flight; `Resuming interrupted copy of database page_cache` is routinely observed on the next boot.
- Heap corruption detected in one case means memory was already corrupt before the abort — worth ruling out that it can occur pre-teardown.

## Suggested fix

1. Wire the signal path to the existing shutdown machinery: on `SIGTERM`/`SIGINT`, run `shutdownWorkers()` and the drain/close sequence already implemented for in-process restarts, then `process.exit()`. Today SIGTERM is a no-op that also suppresses Node's default exit.
2. Close servers and stop accepting new replication work before disposing worker environments, so TLS/fs completions cannot fire into a torn-down environment.
3. Set an explicit `STOPSIGNAL` / stop-timeout for the image that gives the drain enough headroom, and log the shutdown phases so an unclean exit is diagnosable.
4. Given the glibc heap-corruption case, confirm whether the corruption is strictly a teardown artifact or can also occur during normal operation (cf. #288, which sees heap corruption from TLS context churn on a different trigger).

## Related

- #288 — heap corruption / OOM from TLS context churn in `monitorNodeCAs` during reconnect cycles (same family, different trigger)
- #681 — stuck `isIndexing` after a restart; a symptom of workers racing on abrupt restarts

Read the SIGTERM handlers in bin/run.js, utility/processManagement/processManagement.js, and server/threads/threadServer.js, then trace the existing drain path in manageThreads.js. Confirm how shutdownWorkers() and EXTEND_SHUTDOWN_DEADLINE are reached during worker restarts. Done means SIGTERM enters the drain/close sequence and the process exits without the reported TLS or fs completion crashes or stop-timeout SIGKILL.

Written by the indexing model from the issue text.