HarperFast / HarperFast/harper
Compaction record-count validation reports post > pre (likely counting-method mismatch)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
LMDB compaction validation reports a higher record count *after* compaction than before — `total after` should always be ≤ `total before`, but it isn't. Multiple reproductions across Walmart USGM nodes.
## Open questions (from the original report)
- What does `total record count` count? Logical table rows? Raw LMDB keys? Internal/metadata keys?
- Is there a cap applied during the count pass that's different pre vs. post?
- Are we comparing like-for-like? If not, the validation is checking the wrong thing.
- Is this a data-integrity risk, or strictly a reporting / validation bug?
The leading hypothesis based on the round-numbers: pre-compact uses one counting strategy (e.g. capped at a sampled count), post-compact uses a different one (e.g. exact). Need to read the validation code to confirm.
## Acceptance criteria
- Identify which counts are being compared and document them.
- Either: make both counts use the same definition (so the validation is meaningful), or remove the misleading discrepancy warning and replace with whatever check is actually informative.
- Confirm that no actual data is lost or duplicated during compaction (likely a separate verification, not part of this ticket if the discrepancy turns out to be a counting-method mismatch).
## Related
- Compaction is also implicated in [CORE-3008](https://harperdb.atlassian.net/browse/CORE-3008) (Prometheus user wiped during compaction — separate concern).
---
Tracked in Jira: [CORE-2995](https://harperdb.atlassian.net/browse/CORE-2995)
Jira fields to mirror once GH Project is set up: **Feature Type:** *(unset — suggest Tech Debt)* · **Business Impact:** *(unset — suggest Operational efficiency)*
**Target:** V5.2 (Medium, no production impact identified, but worth chasing for correctness of the validation).
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Assessment
This issue has not been assessed yet.