HarperFast / HarperFast/harper
LMDB MDB_NOTFOUND freelist error causing uncaught commit exception under sharded write load
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
## Problem
Under sharding write stress (75k req/s across 5-node cluster), LMDB throws `MDB_NOTFOUND` when attempting to delete a free-space record, which propagates as an `uncaughtException` crashing the process.
```
uncaughtException Error: Commit failed (see commitError for details)
at rejectCommit (lmdb/write.js:612:17)
at resolveWrites (lmdb/write.js:574:40)
...
Error: MDB_NOTFOUND: No matching key/data pair found: Attempting to delete free-space record
at Function. (lmdb/write.js:516:7) {
code: -30798
}
```
## Relation
Closely related to #685 (LMDB freelist SIGSEGV under higher load). Both originate from LMDB freelist corruption under concurrent sharded writes — this ticket surfaces as an application-level `uncaughtException`; #685 surfaces as a native SIGSEGV.
## Reproduction
Performance test repo: https://github.com/HarperDB/sharded-blob-test
Triggered by `multiRightWrites.js` (1 KB records, 5-node sharding, 4.6.0-alpha.3, 75k req/s total).
## Impact
Unhandled commit exception terminates the server. The `commitError` is a pending Promise at the time of rejection, suggesting the LMDB write path doesn't cleanly surface freelist errors to the caller before crashing.
## Next steps
- Investigate alongside #685 — root cause likely the same freelist state corruption
- Determine if `lmdb`'s commit error handling can catch and surface `MDB_NOTFOUND` as a recoverable error rather than crashing
- Evaluate back-pressure / write batching for high-concurrency sharding paths
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Assessment
This issue has not been assessed yet.