HarperFast / HarperFast/harper

read_audit_log batched/full-scan under-counts a hot key's history under concurrent load

Open
#1,855 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

A batched multi-id `read_audit_log(search_type:hash_value, search_values:[...])` call — and separately a full-table audit stream scan — **under-counts** entries for a hot key with a large individual history, under concurrent write load. An **isolated** single-id `read_audit_log(hash_value=[id])` for the same key returns the full, correct count immediately, every time. So the audit data is always present in storage — this is a query-completeness/enumeration bug in the audit-log **read** path under heavy concurrent load + a large combined result set, not write-side data loss.

Reproduced 5/5 on main `f85f41179`, `threads.count:8`, 64-concurrency × 100 rounds (~2500 real RocksDB retries per run).

## Why it matters

A compliance/audit consumer that reads many ids in one batched `read_audit_log` call, or runs a full-table audit scan, can silently get an incomplete result set under load — wrong for an audit log, even though nothing is actually lost on disk. This is a read-path correctness gap in the same family as #1330 and the write-side #1773 fix (which this investigation separately reconfirmed is sound — every flagged id in that probe was acked 2xx with data present, and an isolated audit query always found it; the CI-red there was this same under-enumeration artifact, not a real drop).

## Mechanism (code-search narrowed, not yet instrument-confirmed)

The `hash_value` path (`dataLayer/harperBridge/ResourceBridge.ts:513-532`) is a plain sequential `for..of` over `search_values` calling `table.getHistoryOfRecord(id)` — there's no shared/global result budget across ids (ruled out a "cap" theory). Each `getHistoryOfRecord` (`resources/Table.ts:4444-4479`) walks the record's `previousVersion` chain backward in 100-version windows.

Hypothesis: the batched call is much slower overall (per-id + per-window `await`/`setImmediate` yields), which widens the window during which **concurrent writes to the same hot key mutate the `previousVersion` chain mid-walk** — a read/write race, or a lack of snapshot-consistency, on the audit-history walk. The fast isolated single-id query completes before the race has a chance to bite. To confirm: instrument the window loop (~4453-4477) logging start/`nextVersion`/`highestPreviousVersion` for the hot key in batched vs. isolated runs.

## Bonus: separate confirmed dead-code bug (LOW, worth a cheap co-fix)

The same `getHistoryOfRecord` has an inert window-cap: `const count = 0; do {...} while (count < 1000 && nextVersion)` (~line 4453) — `count` is `const`-0 and never incremented, so the intended 1000-window safety cap never fires; only `nextVersion → 0` terminates the loop. This makes the history walk unbounded for a pathologically large per-key history (perf/DoS-adjacent), which is the opposite direction of the undercount above — a distinct bug, but touches the same loop and is a one-line fix (`let`, or increment `count`).

## Reproduction

```
cd harper && npm run test:integration -- "integrationTests/qa-scratch/qa552b-audit-drop-investigate.test.ts"
```

## Confidence

MEDIUM — reproduced 5/5 under load, mechanism narrowed by source reading but not yet instrumentation-confirmed (the race hypothesis above is the next step if this needs deeper root-causing before a fix).

---
*Surfaced by the QA-explorer exploratory campaign for @kris. Filed by Claude (Sonnet 5) for @kris.*

Contributor guide

Open the contributing guide

Research direction

Start with dataLayer/harperBridge/ResourceBridge.ts:513-532 and resources/Table.ts:4444-4479, then run integrationTests/qa-scratch/qa552b-audit-drop-investigate.test.ts under the stated concurrent load. Instrument the history window loop around lines 4453-4477 to compare batched and isolated reads. Done means batched and full-table reads enumerate the same complete history as isolated reads, with regression coverage for the concurrent case.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.