HarperFast / HarperFast/harper
Normalize the stored audit-reference model: give record version and log identity distinct stored fields
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
On RocksDB a stored record carries two clocks with different jobs — the record's LWW `version`, and the entry's position in the origin node's transaction log (`txnLogKey`) — but the stored format has one slot per role and the code overloads them. #2497 makes the *internal* vocabulary honest and adds the runtime machinery that keeps the two apart; it deliberately does not change the stored format. This issue owns that format change.
## What is overloaded today (at `755329e`)
- **`additionalAuditRefs[].version` holds a log key, not a version.** In the divergent-clock path `Table.ts:3586` pushes `{ version: txnLogKey, nodeId }`, and `isAuditEntryWrite` (`resources/auditStore.ts:386`) matches `ref.version === auditRecord.txnLogKey`. So write identity — the thing that authorizes destroying a tombstone or releasing a blob — is read out of a field named `version`, in a format where a version is legitimately non-unique.
- **`previousVersion` means different things per engine.** `RecordEncoder.ts:1033` stores `isRocksDB ? existingEntry?.version : existingEntry?.localTime`: the record-ordering value on one engine and the log position on the other, in one stored field that the out-of-order history walk reads.
- **`min(version, txnLogKey)` is compatibility machinery, not semantics.** `getAppliedWriteVersion` (`resources/DatabaseTransaction.ts:377`) bounds an applied write's version by its log key purely because an audit-only out-of-order entry can carry the later surviving record version in its body. A format that states the two clocks separately does not need the bound, and carrying it forward would preserve a normalization nobody can justify from the data model.
- **The explicit audit-head reference is a workaround.** #2497 has the primary record retain a reference to its audit head only because, when the clocks diverge, the record has no other way to name its own log position.
## What this issue should produce
1. Distinct stored fields whose names state which clock they are: record version versus `(nodeId, txnLogKey)` write identity.
2. Removal of the audit-only body version, the `min(version, txnLogKey)` bound, and the audit-head reference that exists to compensate for their absence.
3. Coverage of the same-key/same-transaction case. Writes chained to one key inside one transaction share a log key, so `(nodeId, txnLogKey)` is not unique for them. #2412 names this as the write-identity blocker: it has to be resolved — proven to collapse to one durable mutation, or given an intra-transaction ordinal — before identity moves onto the log key wholesale.
4. The two gates #2412 already requires of any format change: a **peer gate** (new field bytes must not reach peers below the format floor; no protocol negotiation exists today) and a **local downgrade gate** (a durable in-store format floor, since an already-released binary cannot be taught to refuse a newer store).
## Why this is separate from the audit key codec
Kept apart from #2518 on purpose. That one changes **key ordering**, so its compatibility risk and likely release sequencing differ from the record/reference format work here; bundling them would put a key-order migration and a record-format migration behind one gate.
Follow-up to #2412.
Contributor guide
Research direction
Start by reading the current storage paths in Table.ts, resources/auditStore.ts, RecordEncoder.ts, and resources/DatabaseTransaction.ts, then review follow-ups #2412 and #2497. Define separate stored fields for record version and write identity, resolve same-key transaction identity, and preserve the peer and local downgrade gates; done means the listed compatibility workarounds and audit-head reference are removed with coverage for the required cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- databases, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100