HarperFast / HarperFast/harper

HTTP/2 (`http.http2: true`) on Fabric fails to apply on restart and breaks replication

Open
#882 0 comments 0 reactions 0 assignees View on GitHub
area:clustering area:security bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Enabling HTTP/2 on a Fabric instance does not work. After setting `http.http2: true` and issuing a restart, the node takes ~6 minutes to come back, the setting **reverts to its original value**, and the cluster ends up with broken replication (a one-directional missing data-db socket) plus `client error - forbidden` when loading logs.

Reported by Jeff Darnton in Slack: https://harperdb.slack.com/archives/C03DLGGN43G/p1780163756639729

## Environment

- Fabric free cluster, multi-node (replication enabled)
- Harper core `main`

## Steps to reproduce

1. On one node, enable HTTP/2: `http.http2: true`.
2. As a precaution, also set the operations API to HTTP/1: `operationsApi.network.http2: false`.
3. Restart the node.

## Observed behavior

- The config change is **accepted** at write time (no validation error).
- After the restart, the container does not come back for **~6 minutes**.
- When it returns, the `http2` setting has **reverted to its original value** (`false`) — the change did not persist/apply.
- `client error - forbidden` when trying to **load logs**, including from a node where HTTP/2 was *not* enabled.
- Replication is broken **one direction only**: a node is missing the data-db socket to one peer, while the reverse direction still has it.

## Expected behavior

Enabling `http.http2: true` should either (a) apply cleanly without breaking replication or operations, or (b) be rejected up front with a clear validation error if HTTP/2 is incompatible with the current configuration — not silently revert after a multi-minute failed restart.

## Investigation notes (confidence-tagged)

**Confirmed (core code):**
- The `http.http2` flag is read at server start and switches the main secure server from the HTTP/1 server (`https.createServer`) to the HTTP/2 server (`http2.createSecureServer`): [`server/http.ts:338`](server/http.ts:338), [`server/http.ts:515`](server/http.ts:515).
- The operations API reads its own flag separately: [`server/operationsServer.ts:348`](server/operationsServer.ts:348).
- Default + JSON schema include the option: [`static/defaultConfig.yaml:12`](static/defaultConfig.yaml:12), [`config-root.schema.json:55`](config-root.schema.json:55).
- There is an explicit note that the operations server avoids HTTP/2 because it breaks ALPN negotiation for the **custom replication protocol**: [`server/http.ts:336-337`](server/http.ts:336).

**Leading hypothesis (medium confidence):** Switching the main secure port to `http2.createSecureServer` breaks the ALPN-negotiated custom replication protocol that runs over that TLS port (the same incompatibility already documented for the operations server). That would explain the lost one-directional data-db socket. A failed/hung startup on the affected node then trips a Fabric health check, which rolls back to last-known-good config (`reverted to original`) and accounts for the ~6-minute restart. The replication transport implementation lives in **harper-pro**, so full root-cause likely spans both repos.

**Secondary gap (confirmed, but not the revert cause):** `http2` (and `keepAliveTimeout`/`timeout`) are absent from the Joi config validator's `http` and `operationsApi.network` schemas ([`validation/configValidator.ts:218-227`](validation/configValidator.ts:218), [`:247-264`](validation/configValidator.ts:247)). Because validation runs with `allowUnknown: true`, the keys are *retained* rather than stripped — so this is not what reverts the setting, but it does mean there is no guardrail to reject HTTP/2 when it is known to be incompatible.

**Lower confidence:** The `forbidden` on log loading is likely a downstream symptom (Studio/Fabric-side auth or an operations API that never finished initializing), not a primary cause.

## Open questions / next steps

- Confirm whether replication runs over the main secure port and whether `http2.createSecureServer` clobbers the replication ALPN protocol (vs. `https.createServer`).
- Decide on the fix shape: make HTTP/2 + replication ALPN coexist on the secure port, OR reject `http.http2: true` at validation when replication is enabled, OR route replication off the HTTP/2 port.
- Add `http2` to the Joi validator schemas so the surface is explicit either way.

cc reporter: Jeff Darnton

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.