HarperFast / HarperFast/harper
LMDB MDB_BAD_VALSIZE / mdb_page_alloc error under sharded blob write load
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
## Problem
During blob write stress tests on a sharded cluster, LMDB fails with `MDB_BAD_VALSIZE` and related page allocation errors, ultimately causing an uncaught commit exception.
Log sequence:
```
Read transaction detected that has been open too long (over 15 minutes), ending transaction
Invalid zero size key
mdb_page_alloc error
mdb_page_touch error
cursor_del0 failed, -30781
```
```
Error: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size: Attempting to delete free-space record
code: -30781
```
## Relation
Part of a cluster of LMDB freelist failures under sharding write load:
- #685 — SIGSEGV from freelist overlap
- #686 — MDB_NOTFOUND on freelist delete
All three share the same reproduction path (sharded-blob-test, `multiRightWrites.js`, multi-node cluster). The stale read transaction noted in logs likely contributes by preventing freelist page recycling, leading to state corruption.
## Reproduction
https://github.com/HarperDB/sharded-blob-test — `multiRightWrites.js` and `singleBlobWrites.js`. Reproduced on both 4.5.9 and 4.6.0-alpha.3.
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Research direction
Start by running the shared reproduction path in sharded-blob-test, especially multiRightWrites.js and singleBlobWrites.js, against the sharded multi-node setup. Trace the stale read transaction and LMDB freelist errors through the Harper write and commit path; done means the reproduction no longer produces the reported MDB_BAD_VALSIZE/page-allocation failures or an uncaught commit exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100