lance-format / lance-format/lance
bug: binding docstrings say stable row ids do not survive an update
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
WriteParams::enable_stable_row_ids (Rust), write_dataset and write_fragments (Python) and CommitBuilder.useStableRowIds (Java) all say the same thing: "These row ids are stable after compaction operations, but not after updates." The format spec says the opposite in docs/src/format/table/row_id_lineage.md: "When stable row IDs are enabled, updates preserve the logical row ID and remap it to a new physical address instead of assigning a new ID." So does the code, and so do the tests: test_row_id_stability_across_update_and_merge_insert asserts the id is unchanged after an update with the feature on, and assert_ne! with it off.
A user reading the docstring concludes they cannot store _rowid externally on a table that takes updates, which is the main thing the feature exists to allow.
Two more places carry a narrower version of the same claim: Dataset::migrate_to_stable_row_ids and Java's hasStableRowIds mention compaction only. A comment above unremappable_index_coverage in rust/lance/src/dataset/optimize.rs says "Stable row ids keep the addresses across a rewrite", which is backwards: a rewrite is what moves addresses. A test comment in rust/lance/src/dataset/rowids.rs says "Updated fragments get fresh row ids" directly above assertions proving that the id survives and no new id is minted.
Steps to reproduce
// The docstring on this field says the id will not survive the update below.
let write_params = WriteParams { enable_stable_row_ids: true, ..Default::default() };
// ... write, then read _rowid for a row, then update that row ...
// The id is unchanged: rust/lance/src/dataset/rowids.rs::test_row_ids_update
Expected behavior
The binding docs state what the spec, the code and the tests already agree on.
Lance version
13.0.0-beta.4 (main)
Language binding
Rust, Python, Java
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the docstrings for WriteParams::enable_stable_row_ids, write_dataset, write_fragments, CommitBuilder.useStableRowIds, Dataset::migrate_to_stable_row_ids, and Java's hasStableRowIds. Compare them with docs/src/format/table/row_id_lineage.md and the assertions in rust/lance/src/dataset/rowids.rs, including test_row_ids_update. Done means all binding docs and the noted comments consistently describe row IDs surviving updates while physical addresses may change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, rust
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100