lance-format / lance-format/lance
bug: row-address mask cache can serve a previous dataset's allow-list at the same URI
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
RowAddrMaskKey is scoped by {version, restrict_hash}, but the metadata cache it lives in is namespaced by dataset URI alone. A dataset dropped and recreated at the same URI restarts its version history at 1, so a long-lived session can serve the new incarnation the previous incarnation's cached stable-row-id allow-list. Stable row ids restart near 0 in both generations, so every row beyond the old generation's id range silently disappears from every indexed query. No error, no warning, just missing rows.
The sibling key RowIdIndexKey already carries the manifest e-tag for exactly this reason, and RowIdSequenceKey is content-addressed. Only the mask key was left without a generation token.
Reproducer, with one shared Session across both incarnations:
- Write 10 rows with
enable_stable_row_ids: true, thendelete("x = 9"). The dataset reaches version 2 and the mask cache holds the allow-list{0..8}. - Drop the dataset directory.
- Write 20 rows at the same URI, again with stable row ids, then
delete("x = 19"). This incarnation also reaches version 2. - Ask for the deletion mask. It comes back as
{0..8}instead of{0..18}, so ids 9 through 18 are treated as deleted.
Reproduced on main at ebba5814c.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at RowAddrMaskKey and the metadata cache namespace, then compare how RowIdIndexKey carries the manifest e-tag and how RowIdSequenceKey is content-addressed. Reproduce the two dataset incarnations with one shared Session and verify that the second deletion mask includes stable row ids 0 through 18 rather than the previous allow-list.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100