lance-format / lance-format/lance

bug: manifest_scan_batch_size=0 can trust a stale manifest hint

Open
#7,928 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Symptom

When manifest_scan_batch_size is configured as 0, manifest discovery can
silently trust a stale version_hint.json. A writer or reader may then operate
from an old manifest version and observe stale epoch, generation, or cursor
state.

Reproduction

  1. Write shard manifest versions 1, 2, and 3.
  2. Replace the best-effort version hint with version 1.
  3. Construct the manifest reader with manifest_scan_batch_size=0.
  4. Call read_latest().

The current implementation returns version 1 instead of discovering version 3.
The zero-sized 0..batch_size range schedules no HEAD requests, so the forward
scan terminates immediately.

Expected behavior

  • ShardWriter::open rejects manifest_scan_batch_size=0 with a contextual
    InvalidInput error that names the parameter and value.
  • The manifest reader defensively rejects zero even when constructed outside the
    writer configuration path, rather than returning stale state.
  • A regression test covers versions 1 through 3 with a hint rolled back to 1.

Scope

This does not change the default batch size or the manifest file format.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at ShardWriter::open and the manifest reader construction path, then reproduce the rollback from hint version 1 to manifest version 3 with manifest_scan_batch_size=0 and call read_latest(). Done means both paths reject zero with contextual InvalidInput errors and a regression test covers manifest versions 1 through 3 with the stale hint.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.