HarperFast / HarperFast/harper

LMDB MDB_NOTFOUND freelist error causing uncaught commit exception under sharded write load

Open
#686 0 comments 0 reactions 0 assignees View on GitHub
area:storage bug from-jira
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 2h
Merged PRs (30d)
205

Description

## Problem

Under sharding write stress (75k req/s across 5-node cluster), LMDB throws `MDB_NOTFOUND` when attempting to delete a free-space record, which propagates as an `uncaughtException` crashing the process.

```
uncaughtException Error: Commit failed (see commitError for details)
at rejectCommit (lmdb/write.js:612:17)
at resolveWrites (lmdb/write.js:574:40)
...
Error: MDB_NOTFOUND: No matching key/data pair found: Attempting to delete free-space record
at Function. (lmdb/write.js:516:7) {
code: -30798
}
```

## Relation

Closely related to #685 (LMDB freelist SIGSEGV under higher load). Both originate from LMDB freelist corruption under concurrent sharded writes — this ticket surfaces as an application-level `uncaughtException`; #685 surfaces as a native SIGSEGV.

## Reproduction

Performance test repo: https://github.com/HarperDB/sharded-blob-test

Triggered by `multiRightWrites.js` (1 KB records, 5-node sharding, 4.6.0-alpha.3, 75k req/s total).

## Impact

Unhandled commit exception terminates the server. The `commitError` is a pending Promise at the time of rejection, suggesting the LMDB write path doesn't cleanly surface freelist errors to the caller before crashing.

## Next steps

- Investigate alongside #685 — root cause likely the same freelist state corruption
- Determine if `lmdb`'s commit error handling can catch and surface `MDB_NOTFOUND` as a recoverable error rather than crashing
- Evaluate back-pressure / write batching for high-concurrency sharding paths

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.