HarperFast / HarperFast/harper
HTTP/2 (`http.http2: true`) on Fabric fails to apply on restart and breaks replication
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Enabling HTTP/2 on a Fabric instance does not work. After setting `http.http2: true` and issuing a restart, the node takes ~6 minutes to come back, the setting **reverts to its original value**, and the cluster ends up with broken replication (a one-directional missing data-db socket) plus `client error - forbidden` when loading logs.
Reported by Jeff Darnton in Slack: https://harperdb.slack.com/archives/C03DLGGN43G/p1780163756639729
## Environment
- Fabric free cluster, multi-node (replication enabled)
- Harper core `main`
## Steps to reproduce
1. On one node, enable HTTP/2: `http.http2: true`.
2. As a precaution, also set the operations API to HTTP/1: `operationsApi.network.http2: false`.
3. Restart the node.
## Observed behavior
- The config change is **accepted** at write time (no validation error).
- After the restart, the container does not come back for **~6 minutes**.
- When it returns, the `http2` setting has **reverted to its original value** (`false`) — the change did not persist/apply.
- `client error - forbidden` when trying to **load logs**, including from a node where HTTP/2 was *not* enabled.
- Replication is broken **one direction only**: a node is missing the data-db socket to one peer, while the reverse direction still has it.
## Expected behavior
Enabling `http.http2: true` should either (a) apply cleanly without breaking replication or operations, or (b) be rejected up front with a clear validation error if HTTP/2 is incompatible with the current configuration — not silently revert after a multi-minute failed restart.
## Investigation notes (confidence-tagged)
**Confirmed (core code):**
- The `http.http2` flag is read at server start and switches the main secure server from the HTTP/1 server (`https.createServer`) to the HTTP/2 server (`http2.createSecureServer`): [`server/http.ts:338`](server/http.ts:338), [`server/http.ts:515`](server/http.ts:515).
- The operations API reads its own flag separately: [`server/operationsServer.ts:348`](server/operationsServer.ts:348).
- Default + JSON schema include the option: [`static/defaultConfig.yaml:12`](static/defaultConfig.yaml:12), [`config-root.schema.json:55`](config-root.schema.json:55).
- There is an explicit note that the operations server avoids HTTP/2 because it breaks ALPN negotiation for the **custom replication protocol**: [`server/http.ts:336-337`](server/http.ts:336).
**Leading hypothesis (medium confidence):** Switching the main secure port to `http2.createSecureServer` breaks the ALPN-negotiated custom replication protocol that runs over that TLS port (the same incompatibility already documented for the operations server). That would explain the lost one-directional data-db socket. A failed/hung startup on the affected node then trips a Fabric health check, which rolls back to last-known-good config (`reverted to original`) and accounts for the ~6-minute restart. The replication transport implementation lives in **harper-pro**, so full root-cause likely spans both repos.
**Secondary gap (confirmed, but not the revert cause):** `http2` (and `keepAliveTimeout`/`timeout`) are absent from the Joi config validator's `http` and `operationsApi.network` schemas ([`validation/configValidator.ts:218-227`](validation/configValidator.ts:218), [`:247-264`](validation/configValidator.ts:247)). Because validation runs with `allowUnknown: true`, the keys are *retained* rather than stripped — so this is not what reverts the setting, but it does mean there is no guardrail to reject HTTP/2 when it is known to be incompatible.
**Lower confidence:** The `forbidden` on log loading is likely a downstream symptom (Studio/Fabric-side auth or an operations API that never finished initializing), not a primary cause.
## Open questions / next steps
- Confirm whether replication runs over the main secure port and whether `http2.createSecureServer` clobbers the replication ALPN protocol (vs. `https.createServer`).
- Decide on the fix shape: make HTTP/2 + replication ALPN coexist on the secure port, OR reject `http.http2: true` at validation when replication is enabled, OR route replication off the HTTP/2 port.
- Add `http2` to the Joi validator schemas so the surface is explicit either way.
cc reporter: Jeff Darnton
Contributor guide
Assessment
This issue has not been assessed yet.