HarperFast / HarperFast/harper
Audit-store key codec infers a float64 timestamp from the key's own first byte (0x42), so an out-of-band key round-trips as a different type
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
`transactionKeyEncoder` (`resources/auditStore.ts:64`) writes a `number` key as a bare big-endian float64 and announces nothing. Its `readKey` then decides the key *is* a float64 by testing the key's own first byte for `66` (`0x42`), and otherwise hands the bytes to `ordered-binary`'s generic `readKey`.
That is the same value-shaped presence contract as #2247, relocated from the audit *entry* to the audit *key*. #2499 fixes the entry and deliberately leaves this one alone.
Two directions fail, in both cases silently:
- **A number outside the `0x42` band.** `0x42` as the leading byte of a positive float64 pins the unbiased exponent to 33..48, i.e. magnitude in `[2**33, 2**49)`. Epoch milliseconds sit comfortably inside that band, which is the only reason nothing has broken. `0`, a negative value, `NaN`, `Infinity`, or a small number leads with a different byte, is still written by the number branch, and is read back through `ordered-binary` as some other type entirely.
- **A non-number key whose encoding starts with `0x42`.** `ordered-binary`'s `writeKey` emits a string whose first char code is >= 28 as raw UTF-8 with no type prefix, so a key beginning with `'B'` reads back as a float64.
**Neither is reachable from any producer in the tree today** — every audit key written is a timestamp inside the band, and the store's key domain is timestamps only. This is filed as a latent format hazard, not a live defect. Two reasons it is worth its own issue rather than a note:
1. It is the last place the #2247 contract still lives after #2499.
2. It gates the stage-2 format work in #2412. Any change here also changes **key ordering**, so it needs its own compatibility and migration design and cannot ride along with an entry-format change.
## Why a cheap guard is not the fix
`writeKey` runs on every audit key, so a warn-only range check adds per-key work on the write path without preventing an unreadable key from being written. Rejecting the write at the codec boundary — what #2499 chose for the entry — is available and is the cheap half. The real fix is an announced encoding, and re-encoding the key changes the sort order of the audit store and of every key already written. That is a migration, not a patch, which is why the two halves should be decided together here rather than piecemeal.
## Scope
- Decide the target key encoding: announced by a prefix, or constrained by fail-closed validation at the writer.
- A migration/compat plan for keys already written and for the ordering guarantees the audit store's range scans and resume cursors depend on.
- Out-of-band-value rejection or normalization at the writer, whichever the above settles on.
Out of scope: the audit *entry* offset defect, which is #2247 / #2499.
Follow-up to #2412.
Related: #2519 (the record/reference format half of the same #2412 follow-up).
Contributor guide
Research direction
Read resources/auditStore.ts around transactionKeyEncoder at line 64, including writeKey and readKey, then compare the related decisions in #2499 and the format work in #2412. Establish whether keys should use an announced encoding or fail-closed validation, and document migration compatibility plus ordering effects on range scans and resume cursors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100