HarperFast / HarperFast/harper

read_audit_log silently under-counts on the paginated (limit) path — group-boundary entry consumed but never yielded

Open
#2,395 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## What happens

`read_audit_log` silently **under-counts on the paginated (`limit`) path only**. Measured: 228 of 12,584 entries missing at `limit=7`, and the loss scales inversely with page size (≈1.8% at limit=7, ≈11.9% at limit=1) — consistent with **one entry dropped per page boundary**. The full-table scan (no `limit`) returns the exact count; only pagination loses rows. This is distinct from harper#1855, which claims the *full scan itself* under-counts a hot key under concurrent load — the opposite signature.

## Mechanism (on `origin/main` @ e16d9616)

`dataLayer/harperBridge/ResourceBridge.ts:773-800` (`groupRecordsInHistory`):

```js
if (enqueued) {
yield enqueued;
count++;
if (limit && limit <= count) { enqueued = undefined; break; }
}
enqueued = { ...new group from current entry... };
```

The entry that *closes* a group (the first entry of the next group, a new distinct timestamp) is consumed from `table.getHistory()` to detect the boundary but is only stored into `enqueued` **after** the `limit` break — so when the page fills, the loop breaks and that boundary entry is discarded, never yielded. Because each `read_audit_log` call re-runs the generator fresh from `start`/`end` with no server-side cursor (`ReadAuditLogObject.ts` has no cursor field; the MCP tool schema at `components/mcp/tools/schemas/operations.ts:239` doesn't even expose `limit`), the dropped boundary entry is lost permanently on every page transition — more boundaries (smaller `limit`) → proportionally more loss.

## Repro

`read_audit_log` a table with a known history count, once unpaginated and once with a small `limit`, page through to the end, and compare totals. Unpaginated matches; paginated is short by ≈(number of page boundaries).

## Suggested fix

Carry the boundary entry across the `limit` break (stash it as the first entry of the next page) instead of discarding it, or thread a real resumable cursor so a page boundary doesn't consume-and-drop.

## Existing work checked

harper#1855 (OPEN) — different, contradictory mechanism (full-scan under-count under concurrent write load; its repro says an isolated read always finds the full history). Not a dup. Searched "audit log pagination cursor" / "read_audit_log limit page" — no hits. Note: `read_audit_log` is marked "Deprecated in favor of `read_transaction_log`" (`operationDescriptions.ts:105`) — relevant to severity framing, but it's a compliance-facing read and the loss is silent.

## What this does not prove

Whether `read_transaction_log` (the recommended replacement) shares the paging path; behavior at very large histories where multiple boundaries fall in one page.

---
*From dispatch QA finding qa-wave-2026072123, verified against harper origin/main e16d9616, 2026-08-30.*

Contributor guide

Open the contributing guide

Research direction

Start in dataLayer/harperBridge/ResourceBridge.ts at groupRecordsInHistory and inspect ReadAuditLogObject.ts plus components/mcp/tools/schemas/operations.ts:239. Reproduce the issue by comparing unpaginated and small-limit read_audit_log totals across pages, then verify that paginated results retain every entry and match the full-table count.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
backend, database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.