HarperFast / HarperFast/harper

Compaction record-count validation reports post > pre (likely counting-method mismatch)

Open
#596 0 comments 0 reactions 0 assignees View on GitHub
area:storage bug from-jira
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

LMDB compaction validation reports a higher record count *after* compaction than before — `total after` should always be ≤ `total before`, but it isn't. Multiple reproductions across Walmart USGM nodes.

## Open questions (from the original report)

- What does `total record count` count? Logical table rows? Raw LMDB keys? Internal/metadata keys?
- Is there a cap applied during the count pass that's different pre vs. post?
- Are we comparing like-for-like? If not, the validation is checking the wrong thing.
- Is this a data-integrity risk, or strictly a reporting / validation bug?

The leading hypothesis based on the round-numbers: pre-compact uses one counting strategy (e.g. capped at a sampled count), post-compact uses a different one (e.g. exact). Need to read the validation code to confirm.

## Acceptance criteria

- Identify which counts are being compared and document them.
- Either: make both counts use the same definition (so the validation is meaningful), or remove the misleading discrepancy warning and replace with whatever check is actually informative.
- Confirm that no actual data is lost or duplicated during compaction (likely a separate verification, not part of this ticket if the discrepancy turns out to be a counting-method mismatch).

## Related

- Compaction is also implicated in [CORE-3008](https://harperdb.atlassian.net/browse/CORE-3008) (Prometheus user wiped during compaction — separate concern).

---

Tracked in Jira: [CORE-2995](https://harperdb.atlassian.net/browse/CORE-2995)
Jira fields to mirror once GH Project is set up: **Feature Type:** *(unset — suggest Tech Debt)* · **Business Impact:** *(unset — suggest Operational efficiency)*
**Target:** V5.2 (Medium, no production impact identified, but worth chasing for correctness of the validation).

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.