lance-format / lance-format/lance
_rowaddr and _rowid not exposed for `merge_insert`
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Sort of a follow up on #3251, I noticed that _rowid and _rowaddr doesn't seem to be usable for merge_insert, while it works for merge. When I try to use it with a subcol update, something like
import pyarrow as pa
import polars as pl
initial_data = pa.table(
{
"a": range(10),
"b": range(10),
"c": range(10, 20),
}
)
dataset = lance.write_dataset(
initial_data, "/tmp/lance/test2.lance"
)
new_values = pl.from_arrow(dataset.to_table(with_row_id=True)).select(pl.col("_rowid"), pl.col("a") * 2)
(dataset.merge_insert("a").when_matched_update_all().execute(new_values))
gives me
OSError: Append with different schema: fields did not match, missing=[b, c], unexpected=[_rowid], location: /Users/runner/work/lance/lance/rust/lance-core/src/datatypes/schema.rs:142:27
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Python repro using dataset.merge_insert("a") and compare its handling of _rowid and _rowaddr with merge. The reported schema mismatch points to lance/rust/lance-core/src/datatypes/schema.rs:142;27. Done means the merge_insert operation can use the row identifiers without the reported missing and unexpected-field error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100