lance-format / lance-format/lance

bug: reject mismatched shard spec before epoch claim

Open
#7,945 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Problem

ShardManifestStore::claim_epoch accepts an incoming shard_spec_id that conflicts with the identity stored in an existing shard manifest.

Under the current writer model, 0 identifies a manually managed shard and 1 is the sole automatic sharding spec. Both are immutable identities; 0 is not a wildcard.

flowchart LR
    A["Stored manifest<br/>automatic spec=1, version=1, epoch=0"]
    B["claim_epoch(manual spec=0)"]
    C["Current result<br/>Ok, spec=1, version=2, epoch=1"]
    D["Expected result<br/>InvalidInput, manifest unchanged"]
    A --> B --> C
    B -.-> D

The existing-manifest branch increments the version and writer epoch while retaining the stored spec through the struct update. The caller therefore receives success even though its configuration disagrees with the returned manifest. Advancing the epoch can also fence a healthy incumbent writer before the invalid claim is rejected.

Reproduction

  1. Initialize an automatically sharded manifest with shard_spec_id=1.
  2. Call claim_epoch(0) as if it were a manually managed shard.
  3. Observe success with a new manifest whose shard_spec_id remains 1, while its version and writer epoch have advanced.

The inverse conflict (stored=0, incoming=1) has the same bug.

Expected behavior

  • Reject an incoming/stored shard spec mismatch with InvalidInput before any manifest write.
  • Include the shard ID and both spec IDs in the error.
  • Leave the manifest version, writer epoch, status, and field values unchanged.
  • Revalidate the identity after version conflicts so concurrent 0/1 claims cannot bypass the check.
  • Preserve successful claims for new shards and matching specs.

Scope

This issue enforces the current manual 0 / automatic 1 defensive invariant. It does not change shard-spec assignment, require monotonic IDs, define spec revisions or activation, or introduce wildcard semantics for shard_spec_id=0.

Implemented independently by #7949. #8112 is complementary writer-side resolution, not a prerequisite.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at ShardManifestStore::claim_epoch, focusing on the existing-manifest and version-conflict branches. Verify that mismatched shard specifications are rejected before writes and that matching and new-shard claims still succeed; confirm the manifest remains unchanged after an invalid claim.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.