HarperFast / HarperFast/harper

Support for deployments larger than 2GB

Open
#524 1 comment 0 reactions 0 assignees View on GitHub
duplicate
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Problem

Payload-based `deploy_component` is capped at ~2 GB. The CLI (`bin/cliOperations.js`) calls `packageDirectory()`, which returns the tar+gzip as a `Buffer`, CBOR-encodes the whole request, and POSTs it. The server (`server/operationsServer.ts`, body limit `REQ_MAX_BODY_SIZE` ~1 GB) reads the body whole before dispatch. Node.js's hard 2 GB Buffer ceiling is the absolute upper bound, and the configured body limit is lower in practice.

Replication compounds the problem: `server.replication.replicateOperation()` relays operations as single payloads over the inter-node WebSocket. Multi-GB payloads exceed both buffer and frame-size limits there as well.

## Plan

### 1. CLI — stream the package as `multipart/form-data`

`bin/cliOperations.js` switches `deploy_component` from CBOR-encoded body to `multipart/form-data`:

- One field part per operation property (`operation`, `project`, `package`, flags, etc.) serialized as JSON.
- One file part carrying the tar+gzip payload, piped from a streaming variant of `packageDirectory()`. No intermediate `Buffer`.
- Sent via Node `https` with `Transfer-Encoding: chunked` — no `Content-Length` precomputation.

Touches `bin/cliOperations.js`, `components/packageComponent.ts` (expose streaming variant), and `utility/common_utils.js` (multipart-aware `httpRequest`).

### 2. Operations API server — receive multipart and stream into extraction

`server/operationsServer.ts` accepts a multipart body when `Content-Type: multipart/form-data`:

- Field parts are decoded into the same `req` object shape the JSON/CBOR path produces.
- The file part is exposed as a `Readable` and passed straight into `deployComponent`, replacing `req.payload: Buffer` with `req.payload: Readable` in this path.
- The existing extraction pipeline in `components/Application.ts:217` (`gunzip-maybe` → `tar-fs.extract`) is already stream-native — it accepts the file part directly, eliminating the in-memory hop.
- `REQ_MAX_BODY_SIZE` is bypassed on the multipart deploy path (the limit only meaningfully constrains the small field parts; the file part is streamed and bounded by disk only).

Touches `server/operationsServer.ts`, `server/serverHelpers/serverHandlers.js`, `server/serverHelpers/serverUtilities.ts`, `components/operations.js`, `components/Application.ts`.

### 3. Replication — direct HTTPS relay instead of WebSocket-framed operation

WebSocket-framed replication cannot carry multi-GB payloads. Replicated `deploy_component` (and `restart_service`, for symmetry) switch to a direct-relay scheme:

1. Origin completes local receive: streams the file part to a temporary tarball on disk while extraction runs (or pipes through a `PassThrough` to a temp file in parallel). This temp file is the source for all peer relays so the CLI uploads once.
2. Origin calls `create_authentication_tokens` over the existing replication channel to mint a short-lived (~5 min TTL) operation JWT, scoped to the specific operations needed on each peer (`deploy_component`, `restart_service`).
3. Origin opens a direct HTTPS connection to each peer's operations API (default port 9925) using that JWT and streams the same `multipart/form-data` request the CLI sent. Peers process it as a normal local deploy with replication suppressed (no re-fanout).
4. Per-peer semantics: peer failures do **not** abort other peers. Origin retries each peer a small number of times on transient transport errors (connection reset, TLS handshake, 502/503) but does **not** retry on application errors (npm install failure, validation, auth). Final result is a per-peer status grid surfaced to the CLI via SSE (#526); the CLI exits non-zero if any peer ultimately failed.
5. Temp tarball is deleted on completion (success or failure).

Open questions / things to nail down during implementation:

- JWT scope mechanics — does `create_authentication_tokens` already support per-operation scoping at sufficient granularity, or do we need a new scope dimension? Audit this before relying on it.
- Peer addressing — reuse the replication peer configuration (host/port/TLS settings) rather than rediscovering nodes. Confirm the operations API port and TLS posture are consistently configured across pro deployments.
- Coexistence with mTLS-proxied operations APIs: the JWT path must work where peers expect client certs too.

Touches the replication module under `replication/` (in harper-pro) and `server/Server.ts`'s `replication` interface — `sendOperationToNode()` currently throws ''Replication not implemented'' (see `server/Server.ts:96`); this work fills it in for the deploy/restart case.

### 4. Backward compatibility

The existing CBOR/JSON request path stays unchanged for small payloads and external operations API clients. Multipart is opt-in on the CLI side and selected automatically for `deploy_component` (we may keep CBOR for tiny payloads, or just unify on multipart — to be decided when measuring overhead).

## Files touched (approximate)

- `bin/cliOperations.js`, `components/packageComponent.ts`, `utility/common_utils.js`
- `server/operationsServer.ts`, `server/serverHelpers/serverHandlers.js`, `server/serverHelpers/serverUtilities.ts`
- `components/operations.js`, `components/Application.ts`
- `server/Server.ts` and the replication module (harper-pro `replication/`)
- New helper: a small multipart parser / formatter shared by CLI and server (likely under `utility/`)

## Related

- #526 — progress reporting / visibility. The SSE channel defined there is what surfaces per-peer relay status to the CLI for this design.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.