lance-format / lance-format/lance

bug: binding docstrings say stable row ids do not survive an update

Open
#9,327 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

WriteParams::enable_stable_row_ids (Rust), write_dataset and write_fragments (Python) and CommitBuilder.useStableRowIds (Java) all say the same thing: "These row ids are stable after compaction operations, but not after updates." The format spec says the opposite in docs/src/format/table/row_id_lineage.md: "When stable row IDs are enabled, updates preserve the logical row ID and remap it to a new physical address instead of assigning a new ID." So does the code, and so do the tests: test_row_id_stability_across_update_and_merge_insert asserts the id is unchanged after an update with the feature on, and assert_ne! with it off.

A user reading the docstring concludes they cannot store _rowid externally on a table that takes updates, which is the main thing the feature exists to allow.

Two more places carry a narrower version of the same claim: Dataset::migrate_to_stable_row_ids and Java's hasStableRowIds mention compaction only. A comment above unremappable_index_coverage in rust/lance/src/dataset/optimize.rs says "Stable row ids keep the addresses across a rewrite", which is backwards: a rewrite is what moves addresses. A test comment in rust/lance/src/dataset/rowids.rs says "Updated fragments get fresh row ids" directly above assertions proving that the id survives and no new id is minted.

Steps to reproduce
// The docstring on this field says the id will not survive the update below.
let write_params = WriteParams { enable_stable_row_ids: true, ..Default::default() };
// ... write, then read _rowid for a row, then update that row ...
// The id is unchanged: rust/lance/src/dataset/rowids.rs::test_row_ids_update
Expected behavior

The binding docs state what the spec, the code and the tests already agree on.

Lance version

13.0.0-beta.4 (main)

Language binding

Rust, Python, Java

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the docstrings for WriteParams::enable_stable_row_ids, write_dataset, write_fragments, CommitBuilder.useStableRowIds, Dataset::migrate_to_stable_row_ids, and Java's hasStableRowIds. Compare them with docs/src/format/table/row_id_lineage.md and the assertions in rust/lance/src/dataset/rowids.rs, including test_row_ids_update. Done means all binding docs and the noted comments consistently describe row IDs surviving updates while physical addresses may change.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python, rust
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.