lance-format / lance-format/lance

bug: a legacy Delete reinstates a data file a concurrent Project pruned

Open
#9,217 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

A Delete that rebases over a concurrently committed Project reinstates a data file the projection had pruned. The resulting manifest carries a data file none of whose fields are in the schema, and the field-id watermark goes back up over an id the projection retired.

Reproduction

On main (verified at 8cdffd30e), through the public write paths only:

  1. Create a dataset with columns a, b; then add_columns a third column c, so fragment 0 carries two data files — [a, b] and [c]. Call this version v.
  2. Stage a delete against v with DeleteBuilder::execute_uncommitted.
  3. Commit a Project down to {a, b}. It lands at v+1, correctly pruning the [c] data file: fragment 0 now has one file.
  4. Commit the staged delete against v. It rebases over the projection and lands at v+2.

Observed across the three versions:

version schema field ids fragment 0 data files Manifest::max_field_id
v [0, 1, 2] [[0, 1], [2]] 2
v+1 (project) [0, 1] [[0, 1]] 1
v+2 (delete) [0, 1] [[0, 1], [2]] 2

The [2] file is back, and the watermark is non-monotonic — so a later write can mint field id 2 while a data file already claims it.

Cause

Same mechanism as https://github.com/lance-format/lance/issues/9216: Operation::Delete's apply replaces the fragment entry wholesale from a post-image built at the read version, which still lists the file the projection dropped. check_delete_txn permits a concurrent Project for the same reason it permits a concurrent DataOverlay.

The scan still succeeds — the orphaned file is simply unreachable — so this does not surface as a read error today. It is a correctness hazard for field-id allocation and leaves a file that cleanup will not collect.

Coverage

A failing test asserting the correct behaviour is committed #[ignore]d as io::commit::conflict_matrix::cases::delete_must_not_reinstate_a_pruned_data_file, pointing at this issue. The (Delete, Project) cell is excluded from that module's matrix until this is fixed.

cargo test -p lance --lib conflict_matrix -- --ignored

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Operation::Delete's apply path and check_delete_txn, then run the ignored conflict-matrix test at io::commit::conflict_matrix::cases::delete_must_not_reinstate_a_pruned_data_file. Compare the Delete-over-Project behavior with the documented versions and confirm the (Delete, Project) matrix case passes without restoring the pruned file or increasing Manifest::max_field_id.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.