lance-format / lance-format/lance

feature: add a backwards-compatibility fixture written with stable row ids

Open
#8,854 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Stable row ids are being taken from experimental to generally available, which makes their on-disk representation a durable compatibility contract. There is currently no test proving that contract holds.

test_data/ has 22 fixture directories covering format and index evolution — v0.30.0_pre_created_at for the index created_at field, fri_straddle_pre_6610 for fragment reuse, and version fixtures back to v0.5.9. None of them was written with enable_stable_row_ids=True. So there is no "an older writer produced this, a current reader must still read it correctly" test for:

  • RowIdSequence and its five segment encodings (Range, RangeWithHoles, RangeWithBitmap, SortedArray, unsorted Array), including the EncodedU64Array bitpacking variants
  • RowIdMeta::Inline bytes in DataFragment
  • manifest.next_row_id and FLAG_STABLE_ROW_IDS
  • RowDatasetVersionSequence for _row_created_at_version and _row_last_updated_at_version

The gap matters more than usual here because the encodings are chosen adaptively by a cost model, so a change to that model silently changes which encoding gets written. Nothing today would catch a reader-side regression on an encoding the current writer stopped choosing.

Suggested shape, following the existing convention:

  • A test_data/<version>/datagen.py that asserts the Lance version it was generated with, creates a dataset with enable_stable_row_ids=True, and deliberately produces a spread of segment encodings — an append-only fragment (Range), one with scattered deletions (RangeWithHoles), one with dense deletions (RangeWithBitmap), and a compacted/merged fragment (SortedArray).
  • Rows that have been updated, so _row_created_at_version and _row_last_updated_at_version diverge and the RLE runs are non-trivial.
  • A Rust test using copy_test_data_to_tmp that opens the fixture, asserts _rowid values are exactly what the generator recorded, asserts a take by row id returns the right rows, and runs Dataset::validate().

Related: the storage layout for these sequences is expected to change (they are moving out of fragment metadata), which is precisely why a fixture written before that change should exist first.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the existing test_data fixture directories and their datagen.py conventions, especially v0.30.0_pre_created_at and fri_straddle_pre_6610. Then trace the Rust test pattern using copy_test_data_to_tmp. Done means a stable-row-id fixture covers the requested segment encodings and updated rows, while the reader test verifies recorded _rowid values, row-id take results, and Dataset::validate().

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.