HarperFast / HarperFast/harper
read_audit_log batched/full-scan under-counts a hot key's history under concurrent load
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A batched multi-id `read_audit_log(search_type:hash_value, search_values:[...])` call — and separately a full-table audit stream scan — **under-counts** entries for a hot key with a large individual history, under concurrent write load. An **isolated** single-id `read_audit_log(hash_value=[id])` for the same key returns the full, correct count immediately, every time. So the audit data is always present in storage — this is a query-completeness/enumeration bug in the audit-log **read** path under heavy concurrent load + a large combined result set, not write-side data loss.
Reproduced 5/5 on main `f85f41179`, `threads.count:8`, 64-concurrency × 100 rounds (~2500 real RocksDB retries per run).
## Why it matters
A compliance/audit consumer that reads many ids in one batched `read_audit_log` call, or runs a full-table audit scan, can silently get an incomplete result set under load — wrong for an audit log, even though nothing is actually lost on disk. This is a read-path correctness gap in the same family as #1330 and the write-side #1773 fix (which this investigation separately reconfirmed is sound — every flagged id in that probe was acked 2xx with data present, and an isolated audit query always found it; the CI-red there was this same under-enumeration artifact, not a real drop).
## Mechanism (code-search narrowed, not yet instrument-confirmed)
The `hash_value` path (`dataLayer/harperBridge/ResourceBridge.ts:513-532`) is a plain sequential `for..of` over `search_values` calling `table.getHistoryOfRecord(id)` — there's no shared/global result budget across ids (ruled out a "cap" theory). Each `getHistoryOfRecord` (`resources/Table.ts:4444-4479`) walks the record's `previousVersion` chain backward in 100-version windows.
Hypothesis: the batched call is much slower overall (per-id + per-window `await`/`setImmediate` yields), which widens the window during which **concurrent writes to the same hot key mutate the `previousVersion` chain mid-walk** — a read/write race, or a lack of snapshot-consistency, on the audit-history walk. The fast isolated single-id query completes before the race has a chance to bite. To confirm: instrument the window loop (~4453-4477) logging start/`nextVersion`/`highestPreviousVersion` for the hot key in batched vs. isolated runs.
## Bonus: separate confirmed dead-code bug (LOW, worth a cheap co-fix)
The same `getHistoryOfRecord` has an inert window-cap: `const count = 0; do {...} while (count < 1000 && nextVersion)` (~line 4453) — `count` is `const`-0 and never incremented, so the intended 1000-window safety cap never fires; only `nextVersion → 0` terminates the loop. This makes the history walk unbounded for a pathologically large per-key history (perf/DoS-adjacent), which is the opposite direction of the undercount above — a distinct bug, but touches the same loop and is a one-line fix (`let`, or increment `count`).
## Reproduction
```
cd harper && npm run test:integration -- "integrationTests/qa-scratch/qa552b-audit-drop-investigate.test.ts"
```
## Confidence
MEDIUM — reproduced 5/5 under load, mechanism narrowed by source reading but not yet instrumentation-confirmed (the race hypothesis above is the next step if this needs deeper root-causing before a fix).
---
*Surfaced by the QA-explorer exploratory campaign for @kris. Filed by Claude (Sonnet 5) for @kris.*
Contributor guide
Research direction
Start with dataLayer/harperBridge/ResourceBridge.ts:513-532 and resources/Table.ts:4444-4479, then run integrationTests/qa-scratch/qa552b-audit-drop-investigate.test.ts under the stated concurrent load. Instrument the history window loop around lines 4453-4477 to compare batched and isolated reads. Done means batched and full-table reads enumerate the same complete history as isolated reads, with regression coverage for the concurrent case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, typescript
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100