HarperFast / HarperFast/harper
Request.withNodeAdapter() cannot serve real Node middleware: Next.js 500s, then >16 KB responses hang silently
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
`Request.withNodeAdapter()` (added in #408, `server/serverHelpers/Request.ts`) is meant to let a Harper component hand a request to an arbitrary Node HTTP handler. It has **no in-tree consumers**, and the first real one — the `@harperfast/nextjs` plugin, which needs it so a Next.js app mounted at a `urlPath` receives the mount-stripped URL — cannot use it: the fake `IncomingMessage`/`ServerResponse` it constructs is not faithful enough for real Node middleware. Every page and route 500s, and once the obvious members are patched in, responses larger than 16 KB hang forever with no error.
Confirmed on `main` (`6d725818c`), `v5.2.9` and `v5.1.27` — none of them have `appendHeader` or `_implicitHeader`, and all build request headers with `Object.create(null)`.
## Symptoms, in the order you hit them
1. **`reqHeaders` is `Object.create(null)`** (`Request.ts:195`); Node's real `IncomingMessage.headers` carries `Object.prototype`. Next's compiled `RouteModule.prepare` calls `headers.hasOwnProperty(...)` → `TypeError: e.hasOwnProperty is not a function`. Every page and route 500s.
2. **No `appendHeader()`** (Node 18.3+). Next's app-page runtime calls it → `TypeError: q.appendHeader is not a function`.
3. **No `_implicitHeader()`**. Next's bundled `compression` calls it on every write → `TypeError: this._implicitHeader is not a function`.
4. **Neither `finished` nor `complete`.** `on-finished` treats that as ALREADY finished (`if (isFinished(msg) !== false) defer(listener)`), so the `send` library — which Next uses for `/_next/static/*` — runs its cleanup immediately and destroys the file read stream. **Every static asset request hangs forever with no error.**
5. With 1–4 patched, `curl` works but a browser page load still hangs. **Responses larger than the response `PassThrough`'s 16 KB high-water mark stall**: `res.write()` returns `false`, Node's `Readable.pipe` waits for `'drain'`, and `compression`'s `res.on` override buffers `'drain'` registrations until its gzip stream exists — which never happens, because a no-op `_implicitHeader()` never triggers `on-headers`' `writeHead` patch. Measured: 9.7 KB / 14.4 KB / 16.8 KB chunks complete, 183 KB and 229 KB never finish. Setting `compress: false` in the app's `next.config.ts` makes the identical page load succeed — that is the isolating experiment.
6. Making `_implicitHeader()` flush headers the way Node's `ServerResponse` does clears the stall and exposes the next layer: `compression` engages and the body is emitted as **raw gzip bytes with no `Content-Encoding` header**, and transfers truncate (`curl` exits 18 after 48 KB of a 183 KB chunk). Header capture resolves at the wrong point relative to `on-headers`' contract.
## Root cause
The adapter's response is `Object.assign(new EventEmitter(), {…})` (`Request.ts:244`) with hand-rolled `write` / `end` / `drain` forwarding. Real Node middleware treats a `ServerResponse` as a **`Writable`** and relies on the `_header` / `_implicitHeader` / `writeHead` ordering contract, so patching members one at a time just surfaces the next one — 1→2→3→4→5→6 above is that sequence, not six independent bugs.
## Proposed fix
Build the adapter's response on a real `Writable` — naturally the response `PassThrough` itself — with status/header capture layered over it, rather than an `EventEmitter` with forwarding methods. Keep `withNodeAdapter`'s existing public shape (`Promise<{ status, headers, body }>`).
Every gap above was invisible to `unitTests/server/serverHelpers/Request.test.js`, which drives synthetic handlers. **The test that matters is one that drives real middleware** — Next.js's request handler, or `express` + `compression` + `send` — over a response larger than 16 KB, and asserts the body arrives complete and correctly encoded.
## Partial patch (fixes 1–4, does NOT fix 5–6)
Included as the starting point, and as evidence that 1–4 are one-line gaps while 5–6 are not:
```diff
- const reqHeaders: Record = Object.create(null);
+ const reqHeaders: Record = {};
@@ nodeRes = Object.assign(new EventEmitter(), {
writableFinished: false,
+ finished: false,
socket: this._nodeRequest.socket,
@@
+ appendHeader(name: string, value: string | number | string[]) {
+ if (Array.isArray(value)) {
+ for (const entry of value) capturedHeaders.append(name, String(entry));
+ } else {
+ capturedHeaders.append(name, String(value));
+ }
+ return nodeRes;
+ },
@@ flushHeaders() { flushHeaders(); },
+ _implicitHeader() {},
@@ end(chunk?, encoding?, callback?) {
nodeRes.writableEnded = true;
+ nodeRes.finished = true;
```
## Reproducing
No Harper runtime needed: a plain Node `http` server, harper's `Request` class loaded from `node_modules`, and Next.js's own request handler. From a `@harperfast/nextjs` checkout with a built fixture (`cd fixtures/next-16 && npx next build`), serve requests by constructing a `Request` and calling `withNodeAdapter(nextHandler)`; symptoms 1–4 appear on `curl`, 5–6 need a browser (chromium reporting which requests never finish). Standalone probe scripts plus this patch are kept locally by @kriszyp.
## Blocked on this
- HarperFast/nextjs#61 — mounted Next.js apps receive the un-stripped URL.
- HarperFast/nextjs#62 — the fix. Its five mounted-app assertions are `test.describe.fixme` and stay that way until this lands; the plugin currently falls back to the direct hand-off (and a one-time warning) when `withNodeAdapter` is absent, so on a Harper without it mounted apps remain non-functional rather than regressing.
The plugin also has a verified alternative that needs no adapter at all (`urlParse(request.url, true)` passed to Next's request handler), but it gives up forwarding of middleware-mutated method and headers, which is the whole point of `withNodeAdapter`.
Contributor guide
Assessment
This issue has not been assessed yet.