lance-format / lance-format/lance
bug: manifest_scan_batch_size=0 can trust a stale manifest hint
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Symptom
When manifest_scan_batch_size is configured as 0, manifest discovery can
silently trust a stale version_hint.json. A writer or reader may then operate
from an old manifest version and observe stale epoch, generation, or cursor
state.
Reproduction
- Write shard manifest versions 1, 2, and 3.
- Replace the best-effort version hint with version 1.
- Construct the manifest reader with
manifest_scan_batch_size=0. - Call
read_latest().
The current implementation returns version 1 instead of discovering version 3.
The zero-sized 0..batch_size range schedules no HEAD requests, so the forward
scan terminates immediately.
Expected behavior
ShardWriter::openrejectsmanifest_scan_batch_size=0with a contextual
InvalidInputerror that names the parameter and value.- The manifest reader defensively rejects zero even when constructed outside the
writer configuration path, rather than returning stale state. - A regression test covers versions 1 through 3 with a hint rolled back to 1.
Scope
This does not change the default batch size or the manifest file format.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at ShardWriter::open and the manifest reader construction path, then reproduce the rollback from hint version 1 to manifest version 3 with manifest_scan_batch_size=0 and call read_latest(). Done means both paths reject zero with contextual InvalidInput errors and a regression test covers manifest versions 1 through 3 with the stale hint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100