HarperFast / HarperFast/harper

LMDB SIGSEGV on freelist corruption under high-throughput sharded writes

Open
#685 0 comments 0 reactions 0 assignees View on GitHub
area:storage bug from-jira
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Problem

Under extreme sharding write load (90k req/s across a 5-node cluster, 3× k6 servers at 30k req/s each), LMDB crashes with a SIGSEGV originating from freelist corruption.

Log excerpt preceding the crash:
```
overlapping duplicate entry 37910899
freelist duplicates/overlaps in free list 1959654 0 1959654 1959664 1959654 1978075 1 1 2147483647 2147483647 11 1 1
conflicting freelist to save:
freelist: 30/254: _-5144654069346994375-8751527078568789555 ...
```

Full SIGSEGV stack in `@lmdb/lmdb-linux-x64`:
```
PID 1 received SIGSEGV for address: 0x7f84042b5000
@lmdb/lmdb-linux-x64/node.abi127.glibc.node(+0x9cb91)
@lmdb/lmdb-linux-x64/node.abi127.glibc.node(+0x890ea)
@lmdb/lmdb-linux-x64/node.abi127.glibc.node(+0x8a66e)
@lmdb/lmdb-linux-x64/node.abi127.glibc.node(+0x54d25)
...
```

## Reproduction

Performance test repo: https://github.com/HarperDB/sharded-blob-test

Triggered by `multiRightWrites.js` (1 KB record size, 5-node sharded cluster, 4.6.0-alpha.3).

## Impact

Server process terminates (SIGSEGV) under stress. Occurs at write rates that exceed the freelist's ability to maintain consistency — likely a race or overflow in LMDB's page recycling under concurrent write pressure.

## Next steps

- Investigate whether upgrading `lmdb` (currently 3.5.4) resolves the freelist overlap detection
- Determine if our transaction/write batching strategy can reduce concurrent write pressure on the freelist
- Consider whether the sharding write path needs back-pressure to stay below LMDB's concurrency limits

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.