HarperFast / HarperFast/harper

Live subscription can deliver a CREATE audit record as `type: "delete"` when the entry is already gone

Open
#2,439 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

The live subscription listener can deliver a record's **CREATE** audit record to subscribers as `type: "delete"`, stamped with the create's own version. This is a wrong event rather than a missing one, and it reaches all three live surfaces (MQTT, SSE, WebSocket).

## Mechanism (verified on `origin/main` @ `fb762a365`)

The per-key listener registered via `addSubscription` (from `resources/Table.ts` `subscribe()`; listener body at `Table.ts:4230-4265`) does not trust `auditRecord.type`. For any non-`message`/`reload`/`end_txn` audit record it recomputes the outgoing type from the **current** primary-store state:

```js
const entry: Entry = primaryStore.getEntry(id);
if (entry) {
if (entry.version !== (auditRecord.recordVersion ?? auditRecord.version)) return;
value = entry.value;
type = entry.metadataFlags & INVALIDATED ? 'invalidate' : value ? 'put' : 'delete';
} else {
type = 'delete'; // Table.ts:4255
}
event.version = auditRecord.version; // Table.ts:4262 — the ORIGINAL record's version
```

If `entry` is missing at delivery time — because the record was deleted or evicted after the CREATE committed but before that audit record's turn came up — a `put` (create) audit record is delivered as `type: "delete"` while carrying the create's version.

The window is real, not theoretical: dispatch comes from a batched, deferred audit-log tail. `resources/transactionBroadcast.ts` `notifyFromTransactionData` uses `NOTIFY_BATCH_SIZE = 256` (:138), defers via `setImmediate` (:58), and calls `subscription.listener(recordId, auditRecord, timestamp, beginTxn)` at :222 — so a fast subsequent delete or a TTL eviction can already have removed the entry by the time the create's callback fires.

## Consequence

A subscriber doing event-sourced replay never sees the create and instead sees a delete carrying the create's version — so ordering and identity are both wrong, not merely incomplete. Create-then-fast-delete and create-under-TTL are ordinary workloads, so the trigger is structural rather than exotic.

## Confirming experiment (for whoever picks this up)

Instrument `Table.ts:4248` (`primaryStore.getEntry(id)`) to log when it returns falsy inside this listener, alongside `auditRecord.type`. Then drive create-then-immediate-delete (or create-then-TTL-evict) on the same key under an open collection subscription and watch for `auditRecord.type === 'put'` reaching the `else { type = 'delete' }` branch. That log line, correlated with the delivered event's version equalling the create's version, distinguishes this misclassification from a coincidental legitimate delete.

## Related

Sibling in the same file family: #2311 (subscription replay silently drops events committed after the audit cursor terminates) — that one is a *silent drop* in a different branch; this one is a *wrong event*.

*Found by the qa-explorer QA loop (finding QA-675, flagged there at MEDIUM confidence and documented in the spec rather than asserted); mechanism since confirmed from source. Dup-searched in `harper` and `harper-pro` — nothing tracks it.*

Contributor guide

Open the contributing guide

Research direction

Start in resources/Table.ts at the addSubscription listener around lines 4230-4265, then read resources/transactionBroadcast.ts around notifyFromTransactionData and the deferred batch dispatch. Confirm the race with the create-then-immediate-delete or TTL experiment described in the issue. Done means a CREATE audit record is no longer delivered as a delete, with correct event identity and version across MQTT, SSE, and WebSocket subscriptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.