lance-format / lance-format/lance

merge_insert violates a row-address ordering invariant during a full upsert

Open
#8,282 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

A deterministic dataset_ops replay can make merge_insert violate its internal row-address
ordering invariant during an UpsertFull. A debug build panics with:

Got updated row address that is not in the original batch

The failing dataset uses storage version 2.0 with stable row IDs disabled. Before the failing merge,
it undergoes a longer sequence of index creation, appends, updates, reads, and merge-insert upserts.
The triggering sequence has not yet been reduced to a standalone public-API reproducer.

The merge implementation walks the original batch's row addresses alongside a peekable iterator
of updated row addresses
. When the next updated row address is less than the current original row address, a
debug_assert! reports the violated invariant. With debug assertions disabled, the same branch
instead continues with (0, original_offset). This makes an incorrect merge result a control-flow
risk, but release-mode output has not been verified and this report does not claim observed data
corruption.

This may be related to #7934, which reported the same assertion when a merge reconciled stale
scalar-index row addresses after fragment rewrites. The current input has not been shown to reach
the invariant through the same cause, so the available evidence does not establish that this is a
regression of #7934.

Expected behavior

merge_insert should preserve the row-address ordering invariant. If this state can occur, the
write should handle it explicitly or return an error before committing instead of relying on a
debug-only assertion and continuing with a fallback mapping in release builds.

Lance version

v10.1.0-beta.2 (94cca93d13b1c731afd6660d46ad2f8d4b4345c4)

Language binding

Rust

Environment

Fedora Linux 44, x86_64, in-memory object store, rustc 1.97.1

Logs / traceback
thread 'main' panicked at /data/rust/cargo/git/checkouts/lance-8ddea23c38163eda/94cca93/rust/lance/src/dataset/write/merge_insert.rs:1406:41:
Got updated row address that is not in the original batch
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'main' panicked at src/dataset_ops.rs:1623:41:
step 14, candidate 15: op UpsertFull { keys: [215, 215, 215, 247, 215, 215, 215, 215, 215, 215, 215, 215, 215, 215, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 255, 255, 255, 255, 138, 255, 255, 255, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 255, 255, 255, 255], rotate: 215, dup_first: true, data: DataParams { len: Big, nulls: Heavy, nans: true } }: merge failed: LanceError(IO): task 2303 panicked with message "Got updated row address that is not in the original batch", /data/rust/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with rust/lance/src/dataset/write/merge_insert.rs around lines 1393-1417 and inspect the row-address ordering assertion and release fallback. Reproduce the deterministic dataset_ops sequence from src/dataset_ops.rs, focusing on the step 14 UpsertFull operation. Done means the invariant is handled explicitly or the write returns an error before commit, with coverage for this failure path.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.