lance-format / lance-format/lance
bug: a legacy Delete reinstates a data file a concurrent Project pruned
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
A Delete that rebases over a concurrently committed Project reinstates a data file the projection had pruned. The resulting manifest carries a data file none of whose fields are in the schema, and the field-id watermark goes back up over an id the projection retired.
Reproduction
On main (verified at 8cdffd30e), through the public write paths only:
- Create a dataset with columns
a,b; thenadd_columnsa third columnc, so fragment 0 carries two data files —[a, b]and[c]. Call this versionv. - Stage a delete against
vwithDeleteBuilder::execute_uncommitted. - Commit a
Projectdown to{a, b}. It lands atv+1, correctly pruning the[c]data file: fragment 0 now has one file. - Commit the staged delete against
v. It rebases over the projection and lands atv+2.
Observed across the three versions:
| version | schema field ids | fragment 0 data files | Manifest::max_field_id |
|---|---|---|---|
v |
[0, 1, 2] |
[[0, 1], [2]] |
2 |
v+1 (project) |
[0, 1] |
[[0, 1]] |
1 |
v+2 (delete) |
[0, 1] |
[[0, 1], [2]] |
2 |
The [2] file is back, and the watermark is non-monotonic — so a later write can mint field id 2 while a data file already claims it.
Cause
Same mechanism as https://github.com/lance-format/lance/issues/9216: Operation::Delete's apply replaces the fragment entry wholesale from a post-image built at the read version, which still lists the file the projection dropped. check_delete_txn permits a concurrent Project for the same reason it permits a concurrent DataOverlay.
The scan still succeeds — the orphaned file is simply unreachable — so this does not surface as a read error today. It is a correctness hazard for field-id allocation and leaves a file that cleanup will not collect.
Coverage
A failing test asserting the correct behaviour is committed #[ignore]d as io::commit::conflict_matrix::cases::delete_must_not_reinstate_a_pruned_data_file, pointing at this issue. The (Delete, Project) cell is excluded from that module's matrix until this is fixed.
cargo test -p lance --lib conflict_matrix -- --ignored
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Operation::Delete's apply path and check_delete_txn, then run the ignored conflict-matrix test at io::commit::conflict_matrix::cases::delete_must_not_reinstate_a_pruned_data_file. Compare the Delete-over-Project behavior with the documented versions and confirm the (Delete, Project) matrix case passes without restoring the pruned file or increasing Manifest::max_field_id.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100