HarperFast / HarperFast/harper

REST JSON ingress ignores charset: non-UTF-8 body stored as mojibake (silent); utf-16 body → 500 not 400 (async parse-error escapes sync try/catch)

Open
#1,630 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary
REST JSON ingress **ignores the request `charset` parameter**. A non-UTF-8 JSON body (`Content-Type: application/json; charset=iso-8859-1`) is silently decoded as UTF-8 → **mojibake string corruption stored with no error (204)**. A `charset=utf-16le` body → **500 not 400** (the streaming deserializer's Promise rejection escapes the synchronous `try/catch`). Chunked framing itself is sound.

## Repro (harper `228eacc0f`, raw TCP — fetch/supertest force Content-Length)
- `charset=iso-8859-1`, body `"café ñoño"` → **204 accepted**, stored + read back as `"caf� �o�o"` (U+FFFD per byte), no error signal.
- `charset=utf-16le`, non-ASCII → **500** `SyntaxError` (embedded NULs break JSON), not a clean 4xx.
- (Positive controls: chunked no-Content-Length UTF-8 with a multibyte char split across a chunk boundary → 200 exact; ~500KB irregular chunks → byte-exact; trailing `; boundary=x` param harmlessly ignored.)

## Root causes (code-confirmed)
1. **Silent corruption:** `server/serverHelpers/contentTypes.ts` — `mediaTypes.get('application/json').deserialize = (data) => JSONParse(data)` runs on a raw `Buffer` and never reads `contentType.parameters.charset`. `JSON.parse(buffer)` coerces via `buffer.toString()` = **always UTF-8**, so any declared non-UTF-8 charset is silently ignored; single-byte charsets (Latin-1/ISO-8859-1) produce syntactically-valid JSON with per-byte U+FFFD-corrupted values → no error.
2. **4xx-as-500:** `server/REST.ts` (~L130-140) wraps the deserializer in a synchronous `try/catch`, but `getDeserializer(..., streaming: true)` returns `stream => streamToBuffer(stream).then(deserialize)` — a Promise. The parse `SyntaxError` rejects **asynchronously**, escaping the synchronous catch, and surfaces at the generic outer handler with no `statusCode` → 500. (Any parse error on a streaming body → 500 not 400 by this mechanism.)

## Fix
Honor `parameters.charset` (decode the Buffer with the declared charset before `JSON.parse`, or return 415 for an unsupported charset), and `await` the streaming deserializer inside the try (or attach a `.catch` mapping to 400).

Found via exploratory QA.

— KrAIs 🤖 (exploratory QA, on Kris's behalf)

Contributor guide

Open the contributing guide

Research direction

Start with server/serverHelpers/contentTypes.ts and server/REST.ts around lines 130-140, then reproduce the raw TCP cases described in the issue. Done means declared charsets are honored or rejected with 415, malformed streaming JSON returns 400 rather than 500, and valid UTF-8 and chunked requests retain their current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.