lance-format / lance-format/lance
Dataset row id reuse old value when `enable_stable_row_ids` enabled
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
Hi team, when enable_stable_row_ids enabled for a dataset, without checking out to a new branch, I expect all row ids to be monotonically increasing; but after restore, I found old row ids are reused.
Steps to reproduce
import tempfile
from pathlib import Path
import lance
import pyarrow as pa
with tempfile.TemporaryDirectory() as tmp:
uri = Path(tmp) / "dataset.lance"
ds = lance.write_dataset(
pa.table({"value": [0]}), uri, enable_stable_row_ids=True
)
assert ds.has_stable_row_ids is True
ds = lance.write_dataset(pa.table({"value": [1]}), ds, mode="append")
before = ds.scanner(with_row_id=True).to_table()
print(f"Version {ds.version} before restore:", before.to_pydict())
ds = ds.checkout_version(1)
ds.restore()
print(f"Version {ds.version} after restore:", ds.to_table().to_pydict())
ds = lance.write_dataset(pa.table({"value": [100]}), ds, mode="append")
after = ds.scanner(with_row_id=True).to_table()
print(f"Version {ds.version} after append:", after.to_pydict())
assert before.to_pydict() == {"value": [0, 1], "_rowid": [0, 1]}
assert after.to_pydict() == {"value": [0, 100], "_rowid": [0, 1]} # this is incorrect
Output
Before restore: {'value': [0, 1], '_rowid': [0, 1]}
After restore: {'value': [0]}
After restore and append: {'value': [0, 100], '_rowid': [0, 1]}
Expected behavior
For row ids after restore and append, the row ids should be [0, 2]
Lance version
ede0c330992571050f79b141f447a6254cd70970
Language binding
Python
Environment
No response
Logs / traceback
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the Python reproduction using enable_stable_row_ids, checkout_version, restore, and append. Trace how row IDs are allocated across restore and the subsequent write, then add a regression test showing that the final IDs are [0, 2] rather than [0, 1].
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100