HarperFast / HarperFast/harper

RecordEncoder: 0x42 prefix-less classic-record decode ambiguity keeps resurfacing — write-side guard or raw-put caller audit

Open
#1,762 2 comments 0 reactions 1 assignee Claimed by @kylebernhardy View on GitHub
area:metrics area:storage bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## The ambiguity

The rocksdb decode heuristic in `resources/RecordEncoder.ts` (`decode()`) detects a local-timestamp prefix by first byte `66` (0x42) — but 0x42 is also classic shared-structure **record-id #2**. A **prefix-less** classic record that happens to start with that id is misread as timestamp-prefixed: 8 bytes are stripped and the decode corrupts (observed as `Could not find typed structure N` → `null`). Typed structs live at 0x20–0x3f and never collide, so this only surfaces with classic structures on the rocksdb path.

The heuristic's in-code comment documents the collision and the caller-local escape hatch (`options.noMetadata`), added in b8490dc06 when the **audit store** hit it. It has since been hit a second time — #1549, where agent session rows written via raw `primaryStore.put` produced prefix-less records and the whole agent died to "Unknown session"; the fix switched to Resource-level `.put` so rows carry the versioned prefix.

Two bites, two caller-local fixes. Nothing prevents the next raw-put caller from re-tripping it — this issue tracks a structural answer.

## Call sites to triage (non-test, on main)

Version-less raw `primaryStore.put(id, record)` — prefix-less writes, the exposed shape:

- `agent/session.ts` (×6) — fixed when #1549 lands
- `resources/analytics/write.ts:1161`
- `resources/models/analyticsTable.ts:162`

Version-carrying raw puts (`ResourceBridge.ts:161`, `Table.ts:868`, `Table.ts:958`) stage version metadata and should decode unambiguously, but the audit should confirm.

## Options

1. **Write-side guard** — a store that carries per-record version metadata refuses (or warns loudly on) a prefix-less write. A raw put with no staged version to a versioned store is arguably always a bug; this turns silent corruption into an immediate error at the write site.
2. **Call-site audit + lint** — triage the sites above, then a lint rule or wrapper so raw `primaryStore.put` against versioned stores can't be reintroduced casually.
3. **Decode-side sanity check** — before committing to the 8-byte strip, check the implied float64 is a plausible ms-epoch timestamp. Narrows the collision window; doesn't eliminate it.

(1) is the strongest: it fails at the write, where the bug is, instead of at some later read.

Credit: the second diagnosis and the decode-path analysis are from #1549's writeup.

🤖 Posted by [Claude](https://claude.com/claude-code) on behalf of @heskew

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.