HarperFast / HarperFast/harper

Normalize the stored audit-reference model: give record version and log identity distinct stored fields

Open
#2,519 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 2h
Merged PRs (30d)
205

Description

On RocksDB a stored record carries two clocks with different jobs — the record's LWW `version`, and the entry's position in the origin node's transaction log (`txnLogKey`) — but the stored format has one slot per role and the code overloads them. #2497 makes the *internal* vocabulary honest and adds the runtime machinery that keeps the two apart; it deliberately does not change the stored format. This issue owns that format change.

## What is overloaded today (at `755329e`)

- **`additionalAuditRefs[].version` holds a log key, not a version.** In the divergent-clock path `Table.ts:3586` pushes `{ version: txnLogKey, nodeId }`, and `isAuditEntryWrite` (`resources/auditStore.ts:386`) matches `ref.version === auditRecord.txnLogKey`. So write identity — the thing that authorizes destroying a tombstone or releasing a blob — is read out of a field named `version`, in a format where a version is legitimately non-unique.
- **`previousVersion` means different things per engine.** `RecordEncoder.ts:1033` stores `isRocksDB ? existingEntry?.version : existingEntry?.localTime`: the record-ordering value on one engine and the log position on the other, in one stored field that the out-of-order history walk reads.
- **`min(version, txnLogKey)` is compatibility machinery, not semantics.** `getAppliedWriteVersion` (`resources/DatabaseTransaction.ts:377`) bounds an applied write's version by its log key purely because an audit-only out-of-order entry can carry the later surviving record version in its body. A format that states the two clocks separately does not need the bound, and carrying it forward would preserve a normalization nobody can justify from the data model.
- **The explicit audit-head reference is a workaround.** #2497 has the primary record retain a reference to its audit head only because, when the clocks diverge, the record has no other way to name its own log position.

## What this issue should produce

1. Distinct stored fields whose names state which clock they are: record version versus `(nodeId, txnLogKey)` write identity.
2. Removal of the audit-only body version, the `min(version, txnLogKey)` bound, and the audit-head reference that exists to compensate for their absence.
3. Coverage of the same-key/same-transaction case. Writes chained to one key inside one transaction share a log key, so `(nodeId, txnLogKey)` is not unique for them. #2412 names this as the write-identity blocker: it has to be resolved — proven to collapse to one durable mutation, or given an intra-transaction ordinal — before identity moves onto the log key wholesale.
4. The two gates #2412 already requires of any format change: a **peer gate** (new field bytes must not reach peers below the format floor; no protocol negotiation exists today) and a **local downgrade gate** (a durable in-store format floor, since an already-released binary cannot be taught to refuse a newer store).

## Why this is separate from the audit key codec

Kept apart from #2518 on purpose. That one changes **key ordering**, so its compatibility risk and likely release sequencing differ from the record/reference format work here; bundling them would put a key-order migration and a record-format migration behind one gate.

Follow-up to #2412.

Contributor guide

Open the contributing guide

Research direction

Start by reading the current storage paths in Table.ts, resources/auditStore.ts, RecordEncoder.ts, and resources/DatabaseTransaction.ts, then review follow-ups #2412 and #2497. Define separate stored fields for record version and write identity, resolve same-key transaction identity, and preserve the peer and local downgrade gates; done means the listed compatibility workarounds and audit-head reference are removed with coverage for the required cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
databases, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.