lance-format / lance-format/lance
feature: add a backwards-compatibility fixture written with stable row ids
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Stable row ids are being taken from experimental to generally available, which makes their on-disk representation a durable compatibility contract. There is currently no test proving that contract holds.
test_data/ has 22 fixture directories covering format and index evolution — v0.30.0_pre_created_at for the index created_at field, fri_straddle_pre_6610 for fragment reuse, and version fixtures back to v0.5.9. None of them was written with enable_stable_row_ids=True. So there is no "an older writer produced this, a current reader must still read it correctly" test for:
RowIdSequenceand its five segment encodings (Range,RangeWithHoles,RangeWithBitmap,SortedArray, unsortedArray), including theEncodedU64Arraybitpacking variantsRowIdMeta::Inlinebytes inDataFragmentmanifest.next_row_idandFLAG_STABLE_ROW_IDSRowDatasetVersionSequencefor_row_created_at_versionand_row_last_updated_at_version
The gap matters more than usual here because the encodings are chosen adaptively by a cost model, so a change to that model silently changes which encoding gets written. Nothing today would catch a reader-side regression on an encoding the current writer stopped choosing.
Suggested shape, following the existing convention:
- A
test_data/<version>/datagen.pythat asserts the Lance version it was generated with, creates a dataset withenable_stable_row_ids=True, and deliberately produces a spread of segment encodings — an append-only fragment (Range), one with scattered deletions (RangeWithHoles), one with dense deletions (RangeWithBitmap), and a compacted/merged fragment (SortedArray). - Rows that have been updated, so
_row_created_at_versionand_row_last_updated_at_versiondiverge and the RLE runs are non-trivial. - A Rust test using
copy_test_data_to_tmpthat opens the fixture, asserts_rowidvalues are exactly what the generator recorded, asserts atakeby row id returns the right rows, and runsDataset::validate().
Related: the storage layout for these sequences is expected to change (they are moving out of fragment metadata), which is precisely why a fixture written before that change should exist first.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the existing test_data fixture directories and their datagen.py conventions, especially v0.30.0_pre_created_at and fri_straddle_pre_6610. Then trace the Rust test pattern using copy_test_data_to_tmp. Done means a stable-row-id fixture covers the requested segment encodings and updated rows, while the reader test verifies recorded _rowid values, row-id take results, and Dataset::validate().
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100