HarperFast / HarperFast/harper
Allow system.mdb to be compacted (currently excluded from compaction)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
`system.mdb` is explicitly skipped by compaction, so operators have no path to reclaim space once internal tables (notably `hdb_analytics`) have grown the file.
## Current behavior
In [resources/databases.ts:666](resources/databases.ts), the compaction loop skips any path ending in `system.mdb`:
```ts
if (store.needsDeletion && !path.endsWith('system.mdb')) {
// ... compact ...
}
```
Combined with long retention windows on `hdb_analytics`, customers can end up with `system.mdb` in the 40–60GB range with no supported way to shrink it (see customer nodes in [CORE-3074](https://harperdb.atlassian.net/browse/CORE-3074), 48–58GB on all four nodes).
## Asks
- Investigate the reason `system.mdb` was excluded — assume an intentional concurrency/correctness concern rather than oversight, but confirm.
- Either: enable compaction for `system.mdb` with appropriate safeguards (e.g. quiesce internal writers), **or** provide an alternative reclamation mechanism (offline-compaction CLI, copy-and-replace utility based on `copyDb.ts`, etc.).
- Document operator guidance for shrinking an over-grown `system.mdb`.
## Acceptance criteria
- Operators have a documented, supported path to reduce a bloated `system.mdb` without manual database surgery.
- The chosen approach is safe under typical operating conditions (or clearly requires Harper to be stopped).
## Priority context
Lower priority than the retention configurability change — current strategic direction is RocksDB, which does not require compaction. This is a workaround for customers still on LMDB.
---
Tracked in Jira: [CORE-3074](https://harperdb.atlassian.net/browse/CORE-3074) (a customer escalation)
Related: [CORE-3045](https://harperdb.atlassian.net/browse/CORE-3045) (Create RocksDB migration/copy script based on existing copyDb.ts)
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Assessment
This issue has not been assessed yet.