ozontech / ozontech/seq-db

`.docs` files larger than 4 GiB return wrong documents

Open
#531 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
131
Forks
16
Avg merge
2d 4h
Merged PRs (30d)
11

Description

Description

Document block offsets are 64-bit everywhere in seq-db except the doc block cache, whose keys are 32-bit. When a block is read, its file offset is truncated to 32 bits before it reaches the cache.

As long as a fraction's document file stays under 4 GiB this is invisible. Past that point offsets wrap and reads go wrong in two independent ways:

  • Key collision: two blocks whose offsets differ by exactly 4 GiB share a single cache entry, so a fetch can be served another block's bytes;
  • Wrong read position: this is a regression from commit e68f8149 (#489). Previously the cache loader was a closure that captured the full 64-bit offset, so only the key was truncated and the read itself was still correct;

After the refactor the loader became a method that receives the cache key, so on every cache miss the read now lands at the truncated position -- the wrong region of the file.

Fix

Two ways to fix it: widen the cache key to 64 bits, which is correct but touches every cache registry and doubles key memory. Or key the document cache by block index instead of byte offset, which keeps the key 32-bit since the block count is far below 4 billion.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the document block cache and the loader refactor from commit e68f8149 (#489), focusing on how offsets become cache keys and how reads use them. Compare widening the key with indexing by block, then verify that blocks beyond 4 GiB cannot collide and are read from their full file positions.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.