`.docs` files larger than 4 GiB return wrong documents
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 131
- Forks
- 16
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 11
Description
Description
Document block offsets are 64-bit everywhere in seq-db except the doc block cache, whose keys are 32-bit. When a block is read, its file offset is truncated to 32 bits before it reaches the cache.
As long as a fraction's document file stays under 4 GiB this is invisible. Past that point offsets wrap and reads go wrong in two independent ways:
- Key collision: two blocks whose offsets differ by exactly 4 GiB share a single cache entry, so a fetch can be served another block's bytes;
- Wrong read position: this is a regression from commit e68f8149 (#489). Previously the cache loader was a closure that captured the full 64-bit offset, so only the key was truncated and the read itself was still correct;
After the refactor the loader became a method that receives the cache key, so on every cache miss the read now lands at the truncated position -- the wrong region of the file.
Fix
Two ways to fix it: widen the cache key to 64 bits, which is correct but touches every cache registry and doubles key memory. Or key the document cache by block index instead of byte offset, which keeps the key 32-bit since the block count is far below 4 billion.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the document block cache and the loader refactor from commit e68f8149 (#489), focusing on how offsets become cache keys and how reads use them. Compare widening the key with indexing by block, then verify that blocks beyond 4 GiB cannot collide and are read from their full file positions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100