lance-format / lance-format/lance
merge_insert violates a row-address ordering invariant during a full upsert
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
A deterministic dataset_ops replay can make merge_insert violate its internal row-address
ordering invariant during an UpsertFull. A debug build panics with:
Got updated row address that is not in the original batch
The failing dataset uses storage version 2.0 with stable row IDs disabled. Before the failing merge,
it undergoes a longer sequence of index creation, appends, updates, reads, and merge-insert upserts.
The triggering sequence has not yet been reduced to a standalone public-API reproducer.
The merge implementation walks the original batch's row addresses alongside a peekable iterator
of updated row addresses. When the next updated row address is less than the current original row address, a
debug_assert! reports the violated invariant. With debug assertions disabled, the same branch
instead continues with (0, original_offset). This makes an incorrect merge result a control-flow
risk, but release-mode output has not been verified and this report does not claim observed data
corruption.
This may be related to #7934, which reported the same assertion when a merge reconciled stale
scalar-index row addresses after fragment rewrites. The current input has not been shown to reach
the invariant through the same cause, so the available evidence does not establish that this is a
regression of #7934.
Expected behavior
merge_insert should preserve the row-address ordering invariant. If this state can occur, the
write should handle it explicitly or return an error before committing instead of relying on a
debug-only assertion and continuing with a fallback mapping in release builds.
Lance version
v10.1.0-beta.2 (94cca93d13b1c731afd6660d46ad2f8d4b4345c4)
Language binding
Rust
Environment
Fedora Linux 44, x86_64, in-memory object store, rustc 1.97.1
Logs / traceback
thread 'main' panicked at /data/rust/cargo/git/checkouts/lance-8ddea23c38163eda/94cca93/rust/lance/src/dataset/write/merge_insert.rs:1406:41:
Got updated row address that is not in the original batch
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at src/dataset_ops.rs:1623:41:
step 14, candidate 15: op UpsertFull { keys: [215, 215, 215, 247, 215, 215, 215, 215, 215, 215, 215, 215, 215, 215, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 255, 255, 255, 255, 138, 255, 255, 255, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 207, 255, 255, 255, 255], rotate: 215, dup_first: true, data: DataParams { len: Big, nulls: Heavy, nans: true } }: merge failed: LanceError(IO): task 2303 panicked with message "Got updated row address that is not in the original batch", /data/rust/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with rust/lance/src/dataset/write/merge_insert.rs around lines 1393-1417 and inspect the row-address ordering assertion and release fallback. Reproduce the deterministic dataset_ops sequence from src/dataset_ops.rs, focusing on the step 14 UpsertFull operation. Done means the invariant is handled explicitly or the write returns an error before commit, with coverage for this failure path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100