HarperFast / HarperFast/harper

Audit log silently has a coverage start-boundary on any node that joined via base-copy - read_audit_log returns clean empty, describe still reports audit: true

Open
#2,212 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

On a clustered instance with `logging.auditLog: true`, `read_audit_log` returns a clean, well-formed **empty** result (HTTP 200, `{"": []}`) for records that verifiably exist and were verifiably written — with no indication that the queried node simply has no audit history for them. The cause is the bulk base-copy path: records applied via base-copy/resync are committed with audit explicitly disabled, by design.

Verified against `harper@5.2.0` (npm):

- `resources/Table.ts:2368` — `const isCopyApply = options?.isCopyApply === true && isRocksDB;`
- `resources/Table.ts:2853` — commit passes `isCopyApply ? false : audit`
- `resources/Table.ts:2881` — `write.blobsAuditReferenced = Boolean(isCopyApply ? false : audit);`

The surrounding comments explain the intent (durability for copied rows comes from the RocksDB flush gating the copy-resume cursor, and the post-copy replay re-delivers writes from `copyStartTime` forward). That replay by construction cannot reconstruct history **predating** the receiving node's own join/resync. Net effect: any node that entered the cluster via base-copy permanently has zero audit history for pre-copy record state.

## Why this is a problem even though it's by design

The gap is invisible at every observation point:

- `describe_all` / `describe_table` still reports `audit: true` for the table — a per-node, per-record coverage fact is presented as a stable per-table capability.
- `read_audit_log` returns a well-formed empty array — indistinguishable from "this record was never touched". It only errors when audit is disabled entirely (`dataLayer/readAuditLog.ts` — `To use this operation audit log must be enabled in harperdb-config.yaml`), so the operator's one signal ("is audit working?") passes while coverage silently has a hard start boundary.

An operator using the audit log for compliance or forensics on any node that ever resynced gets confidently wrong answers.

## Repro sketch

1. Two-node cluster, `logging.auditLog: true` on both, any audited table.
2. Write records on node A over some period.
3. Add (or remove and rejoin) node B so it performs a full base-copy resync.
4. On node B: `read_audit_log` (`search_type: hash_value`) for a pre-copy record id → `200 {"": []}`, while node A returns the full history for the same id, and `describe_table` on B still reports `audit: true`.

## Ask

Any of these would close the gap:

1. Per-table (or per-node) **audit coverage marker** — e.g. `auditCoverageSince: ` recorded at base-copy completion, surfaced in `describe_table` and/or in the `read_audit_log` response envelope.
2. `read_audit_log` distinguishing "no entries, full coverage" from "no entries, history may predate this node's audit coverage".
3. At minimum: documenting the boundary on the audit-log docs page so operators know a resynced node is not an audit source of record.

## Footnote: config doc drift

`config-root.schema.json:332` describes `logging.auditLog` as "Default: false", but `static/defaultConfig.yaml:43` (which seeds new installs via `config/configUtils.ts`) ships `auditLog: true`. Whichever is intended, the two disagree.

Contributor guide

Open the contributing guide

Research direction

Start by reading resources/Table.ts around the isCopyApply commit path, then inspect dataLayer/readAuditLog.ts and the describe_table response handling. Confirm the pre-copy audit boundary and decide which of the proposed coverage signals is appropriate, including the config-root.schema.json and static/defaultConfig.yaml mismatch. Done means operators can distinguish complete audit coverage from a resynced node with missing history, with behavior verified by focused tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.