HarperFast / HarperFast/harper

Enable compression for file-backed blobs (opt-in, per-content-type), and stream-inflate on read

Open
#2,443 0 comments 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Harper's file-backed blobs — anything at or above the 8 KB inline threshold, stored under `storage.blobPaths` — are written to disk **uncompressed, always.** Record values in RocksDB are lz4-compressed by default (`storage.compression`, applied to both the block codec and `blob_compression_type`), and blobs below the threshold ride along inside the record and inherit that. Everything above it — the bytes that dominate a blob-heavy deployment's footprint — gets nothing.

The write path already exists and is complete. It has simply never been switched on.

## Current state

- `BlobCreationOptions.compress` (`resources/blob.ts:1239`) is the only switch, and no production caller sets it. The sole `compress: true` in the tree is `unitTests/resources/blob.test.js:142`.
- There is no config surface for it at all.
- The HTTP-upload path never gets the chance: an uploaded body becomes a plain `Blob` and is written by `saveBlob` → `writeBlobWithStream` (`resources/blob.ts:1290`), which takes no options argument.

So `storage.compression: true` — the shipped default — silently does not apply to the largest objects Harper stores.

## Scope

### 1. Read path must stream-inflate (prerequisite, not a follow-up)

`resources/blob.ts:604` is **not** range-gated: *any* read of a `DEFLATE_TYPE` blob, including a plain full streaming read, delegates to `blob.bytes()`, which inflates the entire file and enqueues it as one chunk. A 500 MB compressed blob is 500 MB of heap per concurrent read.

This is a latent defect in shipped code — `createBlob` is exported to components (`security/jsLoader.ts:879`), so `createBlob(x, { compress: true })` is reachable from user code today — but it is unhit because nothing enables compression. It must become a streaming `createInflate` before compression is turned on, or this trades disk for OOM.

Ranged reads of a compressed blob then become inflate-and-discard up to `start`: bounded memory, `O(start)` CPU, no random access. That is accepted.

### 2. Config: opt-in, per content type

```yaml
storage:
blobs:
compression: # key absent = off
default: { codec: deflate, threshold: 65536 }
'text/*': { codec: deflate, threshold: 8192 }
'image/*': false
'video/*': false
'application/gzip': false
```

Match precedence: exact type → `type/*` → `default`. A `false` entry is how an incompressible type opts out, so this subsumes a separate skip-list.

`application/gzip: false` has to be a **shipped default**, not something an operator is expected to know: `components/deploymentRecorder.ts:259` writes deployment payload blobs with that content type, and double-compressing them is pure waste.

Needs the corresponding entry in `config-root.schema.json` and `validation/configValidator.ts`.

### 3. Codec: deflate only

Type byte `1` (`DEFLATE_TYPE`, `resources/blob.ts:71`) is the already-shipped identifier — no format extension required. Compatibility, verified against the release branches rather than assumed:

| Reader | `bytes()` (buffered) | `stream()` — what an HTTP `GET` of a blob uses |
|---|---|---|
| v5.0.x | yes | **no type check** — streams the raw body, ends in `Blob is incomplete` |
| v5.1.x | yes | same |
| v5.2.x | yes | yes — the `blob.ts:604` branch landed in v5.2.0 |

v5.0.0 and v5.1.0 carry exactly four `DEFLATE_TYPE` sites (constant, header, `bytes()`, and the write-side OR) and none inside the `ReadableStream` read loop; that loop takes `size` from the header's low 48 bits, which is the *uncompressed* length, so a shorter compressed body never satisfies the completion check.

Net: deflate blobs are fully readable by >= 5.2 and fail **loudly** — never silently served as content — on older readers. This ships on `main` (5.3), which makes 5.2 the realistic downgrade target, so no version fence is needed.

zstd is a deliberate follow-up, not part of this issue. When it lands it must take a **new** type id (`2`) and never redefine `1`:

- Unknown type ids already fail closed — `blobHeaderIndicatesIncomplete()` returns `true` for anything that isn't `UNCOMPRESSED_TYPE` or `DEFLATE_TYPE`, so the repair sweep and copy-capture machinery classify an unrecognized codec as `pending` rather than as content.
- Reusing `1` would hand zstd bytes to `zlib.inflate`. It happens to error (zstd's `28 B5 2F FD` magic fails zlib's FCHECK mod-31 test), but that is a checksum coincidence, and the operator sees a corruption-shaped `Z_DATA_ERROR` on a "deflate" blob for what is really a version mismatch.
- A new codec also owes a read-support-one-release-before-write-support cycle, which deflate does not.

### 4. Preserve the codec across replication (harper-pro companion PR)

The sender currently streams `blob.stream()` — inflated bytes (`harper-pro replication/replicationConnection.ts:4496`). Left alone, a compressed blob would be inflated on the sender and recompressed on the receiver: wasted CPU at both ends and zero bandwidth saving. Pass the compressed body through instead.

Negotiation has to work without a protocol bump, because blob sends are sender-push (there is no receiver-initiated blob request message) and the WS subprotocol is exact-matched (`harperdb-replication-v1`, `replication/replicator.ts:143`), so bumping it would hard-break older peers. The mechanism:

- the follower advertises `acceptBlobCodecs: ['deflate']` on `SUBSCRIPTION_REQUEST` (message type 129);
- the leader stamps `codec` on the `BLOB_CHUNK` (146) `blobInfo` object and streams the raw post-header body **only** when the peer advertised support;
- a peer that advertises nothing gets today's inflated stream. No fence, no version gate, no behavior change for mixed clusters.

Two consequences to implement deliberately:

- the receiver's on-disk codec becomes the **sender's**, ignoring the receiver's own config. That is the intended tradeoff — it is what "no recompress" means.
- the receive side needs a write helper that stamps a known header and writes raw bytes, because `writeBlobWithStream` derives the stored size from `compressedStream.bytesWritten`.

## Decisions already settled

| Question | Decision |
|---|---|
| Default on or opt-in | **Opt-in.** Absent config = off. |
| Codec | **deflate only.** zstd is a separate follow-up with a new type id. |
| Selection policy | **Per-content-type map with a size threshold**, `false` to opt a type out. |
| Ranged reads | **Inflate-and-discard is fine.** No block framing, no offset index. |
| Replication | **Preserve compression on the wire** — negotiated, never inflate-then-recompress. |

## Out of scope

- zstd / brotli codecs.
- A block-framed format with an offset index for random access into compressed blobs.
- Recompressing blobs already on disk. This applies to new writes only; a rewrite/migration path is a separate question.

## Verification worth building

- A blob written with each config shape lands with the expected header type byte, and reads back byte-identical through `bytes()`, `stream()`, and a ranged `stream(start, end)`.
- A large compressed blob streams with bounded heap — the point of the read-path fix, and the thing a test must actually watch fail without it.
- `application/gzip` and `image/*` blobs are stored uncompressed under the shipped defaults.
- Replication between two nodes that both advertise the codec transfers the compressed body (assert bytes on the wire, not just the result), and a node that advertises nothing still converges.
- `isBlobComplete` / the repair sweep still correctly classify a compressed blob, including a torn one.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.