hyperdxio / hyperdxio/hyperdx-js

Session recorder: unbounded `RangeError: Bad value` crash loop on Safari shared `TextDecoder` reused in `emit()` with no error handling

Open
#303 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug external
Dominant language
TypeScript
Stars
25
Forks
30
Avg merge
4h 26m
Merged PRs (30d)
6

Description

Summary

On Safari (WebKit), the session recorder's emit() callback can enter a permanent, unbounded error loop: an uncaught RangeError: Bad value thrown from native TextDecoder.prototype.decode() on every rrweb mutation flush (~40/sec, rAF-throttled), continuing until the page is closed or reloaded. One of our users produced 64,606 uncaught errors over 25 minutes in a single session.

The recorder reuses a single module-level TextDecoder for the life of the page, and WebKit has a known engine bug where a TextDecoder instance degrades after high cumulative decode volume (~2GB) and then throws RangeError: Bad value on valid input, permanently. Because emit() has no try/catch and nothing backs off or stops the recorder on repeated failure, one corrupted decoder turns into an indefinite crash loop.

Environment

  • @hyperdx/browser 0.24.0 (@hyperdx/otel-web 0.18.0 per RUM resource attributes); the relevant code is unchanged on main and in 0.25.1 as of 2026-08-10
  • Safari 18.3, macOS, desktop
  • Long-lived SPA session (~45 min of continuous recording before onset), recording enabled with defaults (maskAllInputs: true, replay not disabled)

Observed behavior (production telemetry)

  • Error: RangeError: Bad value, unhandled, thrown from decode@[native code]
  • Onset ~45 minutes into a heavy-DOM recording session; then sustained ~40 errors/second for 25 minutes, peaking at ~156/second (64,606 total from one user, 92% of our entire onerror volume that day)
  • Rate matches rrweb's rAF-throttled mutation flush; every flush re-enters emit() and re-throws
  • The loop survives SPA route changes (recorder is a module-level singleton) and only stops on page unload
  • Only WebKit affected; Chrome/Firefox users in the same cohort showed zero instances

Stack (minified frames, resolved via sourcemapped rebuild all frames land in @hyperdx/browser/build/index.js):

decode@[native code]
emit@.../assets/index-*.js          <- session-recorder emit() (chunk/decode loop)
x0@.../assets/index-*.js            <- rrweb internal event dispatch
ap@.../assets/index-*.js
(anonymous) x3                      <- rrweb mutation-buffer flush chain
Telemetry Aggregates (per-minute rates, stack variants)
Telemetry aggregates RangeError: Bad value incident (upstream-safe)

Companion data for the hyperdx-js issue. Source: our OpenTelemetry browser RUM
(onerror spans) in ClickHouse, filtered to error.message = 'Bad value' for the
affected session window (2026-08-07, times UTC). All user/session identifiers
removed; stack frames contain only public bundle URLs.

Shape of the incident

  • Single user, single browser session (Safari 18.3, macOS, desktop)
  • 64,606 total uncaught errors, 01:14:29 → 01:40:45 UTC (~26 min)
  • Average ≈ 38 errors/sec sustained; peak 9,392 errors in one minute (~156/sec)
  • Errors continued across SPA route changes within the same page session and
    stopped only at page unload

Errors per minute

Minute (UTC) Errors Minute (UTC) Errors
01:14 503 01:28 7,113
01:15 403 01:29 8,320
01:16 427 01:30 8,944
01:17 370 01:31 9,392
01:18 215 01:32 8,226
01:19 241 01:33 6,489
01:20 288 01:34 517
01:21 314 01:35 333
01:22 137 01:36 453
01:23 598 01:37 586
01:24 1,063 01:38 389
01:25 1,069 01:39 90
01:26 930 01:40 10
01:27 7,214

The rate tracks page activity (higher while the user interacted heavily, lower
while idle) consistent with rrweb emitting per mutation/interaction batch and
every emit throwing.

Stack variants (29 distinct; top 10 shown)

All variants share the same top frames native decode called from the
recorder's emit and differ only in which rrweb observer path entered emit:

Spans Avg/sec Entry path into emit (inferred from lower frames)
59,744 37.9 mutation-buffer flush (rAF-driven; dominant variant)
1,215 0.8 observer path A (timer/callback-queue entry)
1,089 0.7 observer path B (async wrapper entry)
762 0.5 observer path C
457 0.3 observer path D (named handler pair)
404 0.3 observer path A, deeper callback nesting
352 0.2 observer path C, alternate scheduling
153 0.1 observer path B, direct
71 0.1 observer path C, alternate branch
70 1.8 focus@[native code] → UI event handler chain (burst)

Representative dominant stack (minified; all frames resolve to the
@hyperdx/browser vendored chunk in our bundle):

decode@[native code]
emit@.../assets/index-*.js:134:65627
x0@.../assets/index-*.js:116:32277
ap@.../assets/index-*.js:116:32807
@.../assets/index-*.js:116:13017
@.../assets/index-*.js:116:6681
@.../assets/index-*.js:116:3719

Why this supports instance-wide decoder corruption

Ten-plus distinct call paths mutation flush, input/scroll observers, even a
native focus event chain all fail at the same decode call for 26
minutes straight. The input payloads across those paths are unrelated rrweb
event JSONs, so per-payload explanations (a specific bad byte sequence) do not
fit; a corrupted shared TextDecoder instance failing on all input does.

Root cause

packages/session-recorder/src/index.ts (current main):

const MAX_CHUNK_SIZE = 950 * 1024; // 972800 bytes
const encoder = new TextEncoder();
const decoder = new TextDecoder();   // <-- one instance, reused for the page lifetime
...
emit(srcEvent) {
  ...
  const body = encoder.encode(
    ensureStringifiedMaxMessageSize(JSON.stringify(event)),
  );
  const totalC = Math.ceil(body.byteLength / MAX_CHUNK_SIZE);
  for (let i = 0; i < totalC; i++) {
    const start = i * MAX_CHUNK_SIZE;
    const end = (i + 1) * MAX_CHUNK_SIZE;
    const log = convert(decoder.decode(body.slice(start, end)), time, {...});
    //                  ^^^^^^^^^^^^^^ no try/catch; throws propagate into rrweb dispatch
    processor.onLog(log);
  }
},

Three compounding problems:

  1. WebKit decoder corruption (the trigger). WebKit has an open bug where a TextDecoder instance that has decoded a large cumulative volume (~2GB reported) starts throwing RangeError: Bad value on valid input, permanently:

    Every emit() round-trips the entire serialized event through encoder.encode()decoder.decode(), so a long recording session on a busy DOM pushes the shared decoder toward the corruption threshold. Once it corrupts, every subsequent event throws.

  2. No error handling / no back-off (the amplifier). The throw escapes emit() into rrweb's dispatch and surfaces as an uncaught error on every mutation flush. Nothing catches it, counts failures, or stops the recorder so the SDK errors ~40x/sec indefinitely on the affected page.

  3. Chunk-boundary data corruption (related bug, all browsers). For events over MAX_CHUNK_SIZE, each byte-slice is decoded independently without { stream: true }. A slice boundary that bisects a multi-byte UTF-8 character yields U+FFFD replacement characters at the seam of adjacent chunks, silently corrupting the reassembled replay JSON. (Not the cause of the Safari throw, but the same three lines of code.)

Reproduction

The engine bug needs high cumulative decode volume (~2GB per the WebKit reports), so the practical repro is the usage pattern, on Safari:

// Run in the Safari Web Inspector console. Mirrors the recorder's usage:
// one shared decoder, repeated non-streaming decode of ~950KB payloads.
// Caps at 10 GB so it terminates either way. (Async IIFE — Safari's console
// doesn't accept top-level await.)
(async () => {
  const decoder = new TextDecoder();
  const payload = new TextEncoder().encode("x".repeat(950 * 1024));
  const CAP = 10 * 2 ** 30;
  let total = 0;
  try {
    while (total < CAP) {
      decoder.decode(payload);
      total += payload.byteLength;
      if (total % (500 * 1024 * 1024) < payload.byteLength) {
        console.log(`${(total / 2 ** 30).toFixed(2)} GB decoded`);
        await new Promise((r) => setTimeout(r)); // let logs flush
      }
    }
    console.log(`no throw after ${(total / 2 ** 30).toFixed(2)} GB`);
  } catch (e) {
    console.log(`threw after ${(total / 2 ** 30).toFixed(2)} GB:`, e);
    // The key check: a FRESH decoder still works while the old instance is corrupted
    console.log(
      "fresh decoder works?",
      new TextDecoder().decode(payload).length === payload.byteLength,
    );
  }
})();

Per the WebKit / wasm-bindgen reports this starts throwing RangeError: Bad value around the ~2GB mark; afterwards the same instance keeps throwing on input it previously decoded fine, while a freshly constructed TextDecoder works which is what turns the recorder into a permanent crash loop. Real-world trigger: any long recording session on a mutation-heavy page in Safari.

Our repro results so far (transparency): on Safari 26.5.2 (macOS, the newest build we have on hand) this snippet did NOT reproduce — both the fixed-size ASCII variant above and a multi-byte/varied-slice variant ran clean to 10 GB cumulative. The production incident and the linked WebKit/wasm-bindgen reports are all on Safari 17.5–18.x (our user: 18.3), so the engine bug may be fixed in newer Safari — but 18.x remains widely deployed (school/enterprise-managed Macs update slowly), the incident telemetry is unambiguous (decode@[native code] inside emit, 64k throws), and the SDK currently has no defense in depth if any engine misbehaves in this hot path. If you have Safari 18.x available, the snippet should be run there.

Suggested fix

  • Wrap the decoder.decode() call in try/catch; on failure, recreate the decoder (decoder = new TextDecoder()) and retry once. (Safe: the decoder is stateless here since {stream: true} isn't used.)
  • Add a circuit breaker: after N consecutive emit() failures, stop the session recorder instead of erroring on every event forever.
  • Skip the encode→decode round-trip entirely when totalC === 1 (the overwhelmingly common case) the JSON string is already in hand, so most events never need to touch TextDecoder at all. This also drastically slows cumulative-volume decoder aging.
  • For genuinely multi-chunk events, decode with { stream: true } (or slice the string, not the bytes) to fix the U+FFFD boundary corruption.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/session-recorder/src/index.ts and inspect the session recorder's emit() decode loop, especially the shared TextDecoder and MAX_CHUNK_SIZE handling. Run the Safari reproduction if Safari 18.x is available, then verify the recorder no longer produces an uncaught repeated failure and that multi-chunk UTF-8 data is not corrupted at chunk boundaries.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
frontend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.