HarperFast / HarperFast/harper

Audit-store key codec infers a float64 timestamp from the key's own first byte (0x42), so an out-of-band key round-trips as a different type

Open
#2,518 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 2h
Merged PRs (30d)
205

Description

`transactionKeyEncoder` (`resources/auditStore.ts:64`) writes a `number` key as a bare big-endian float64 and announces nothing. Its `readKey` then decides the key *is* a float64 by testing the key's own first byte for `66` (`0x42`), and otherwise hands the bytes to `ordered-binary`'s generic `readKey`.

That is the same value-shaped presence contract as #2247, relocated from the audit *entry* to the audit *key*. #2499 fixes the entry and deliberately leaves this one alone.

Two directions fail, in both cases silently:

- **A number outside the `0x42` band.** `0x42` as the leading byte of a positive float64 pins the unbiased exponent to 33..48, i.e. magnitude in `[2**33, 2**49)`. Epoch milliseconds sit comfortably inside that band, which is the only reason nothing has broken. `0`, a negative value, `NaN`, `Infinity`, or a small number leads with a different byte, is still written by the number branch, and is read back through `ordered-binary` as some other type entirely.
- **A non-number key whose encoding starts with `0x42`.** `ordered-binary`'s `writeKey` emits a string whose first char code is >= 28 as raw UTF-8 with no type prefix, so a key beginning with `'B'` reads back as a float64.

**Neither is reachable from any producer in the tree today** — every audit key written is a timestamp inside the band, and the store's key domain is timestamps only. This is filed as a latent format hazard, not a live defect. Two reasons it is worth its own issue rather than a note:

1. It is the last place the #2247 contract still lives after #2499.
2. It gates the stage-2 format work in #2412. Any change here also changes **key ordering**, so it needs its own compatibility and migration design and cannot ride along with an entry-format change.

## Why a cheap guard is not the fix

`writeKey` runs on every audit key, so a warn-only range check adds per-key work on the write path without preventing an unreadable key from being written. Rejecting the write at the codec boundary — what #2499 chose for the entry — is available and is the cheap half. The real fix is an announced encoding, and re-encoding the key changes the sort order of the audit store and of every key already written. That is a migration, not a patch, which is why the two halves should be decided together here rather than piecemeal.

## Scope

- Decide the target key encoding: announced by a prefix, or constrained by fail-closed validation at the writer.
- A migration/compat plan for keys already written and for the ordering guarantees the audit store's range scans and resume cursors depend on.
- Out-of-band-value rejection or normalization at the writer, whichever the above settles on.

Out of scope: the audit *entry* offset defect, which is #2247 / #2499.

Follow-up to #2412.

Related: #2519 (the record/reference format half of the same #2412 follow-up).

Contributor guide

Open the contributing guide

Research direction

Read resources/auditStore.ts around transactionKeyEncoder at line 64, including writeKey and readKey, then compare the related decisions in #2499 and the format work in #2412. Establish whether keys should use an announced encoding or fail-closed validation, and document migration compatibility plus ordering effects on range scans and resume cursors.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.