lance-format / lance-format/lance

feat: support conditional delete in merge-insert when_matched clause

Open
#7,725 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Problem

WhenMatched supports conditional update through UpdateIf and UpdateIfExpr, but delete is only unconditional. Conditional delete exists solely on WhenNotMatchedBySource::DeleteIf, and the Python binding when_matched_delete() takes no condition.

This makes it impossible to express "delete the matched target row only when a condition holds" in a single merge-insert. Examples are tombstone-driven deletes (source.deleted = true) and stale-delete guards (source.last_update > target.last_update). The current workaround is a separate scan and delete pass, which is not atomic with the merge.

Expected behavior

Rust:

let job = dataset.merge_insert(&["id"])
    .when_matched(WhenMatched::delete_if(&dataset, "source.deleted = true")?)
    ...

Python, mirroring when_matched_update_all(condition):

dataset.merge_insert("id") \
    .when_matched_delete("source.deleted = true") \
    .when_not_matched_insert_all() \
    .execute(new_data)

The condition supports source. and target. qualified references like UpdateIf does. Matched rows failing the condition are left untouched. Bare when_matched_delete() keeps its current unconditional behavior.

Proposed implementation

  • Add WhenMatched::DeleteIf(String) and DeleteIfExpr(Expr) variants mirroring UpdateIf and UpdateIfExpr, with parsing deferred because the two execution paths resolve source. and target. references against different schemas (relation-qualified columns on the standard plan, combined-schema struct fields on the indexed path).
  • Fast path: new CASE arms in assign_action.rs emitting Action::Delete when a row is matched and the condition is true, falling through to Action::Nothing otherwise. Delete-only plans keep using DeleteOnlyMergeInsertExec unchanged since the condition is baked into the action assignment.
  • Slow (indexed) path: reuse the Merger's existing match_filter_expr compilation and apply the mask to matched rows before recording deletions.
  • Python: optional condition parameter on the existing when_matched_delete, fully backward compatible.

Suggested coverage

  • Rust tests for both execution paths (the fast-path plan and the indexed Merger path), asserting num_deleted_rows counts only condition-true matches and surviving rows are unmodified. Error case for non-boolean conditions.
  • Python test extending test_merge_insert_when_matched_delete with conditional cases.

Follow-up (out of scope here): Java MergeInsertParams.withMatchedDeleteIf(String) mirroring withNotMatchedBySourceDeleteIf.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the WhenMatched variants, assign_action.rs, and the existing Merger match_filter_expr path; compare the fast-path plan with the indexed path. Extend the Rust coverage for both paths and the non-boolean error, then extend test_merge_insert_when_matched_delete for Python. Done means conditional matches alone are deleted, other matched rows remain unchanged, and unconditional deletion still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.