lance-format / lance-format/lance
`Operation::Rewrite`'s `frag_reuse_index` is dropped when the transaction is serialized, so the deferred-remap conflict exemption never applies to a competing transaction
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
Operation::Rewrite carries an optional frag_reuse_index, and the conflict resolver uses its presence to exempt a compaction from conflicting with a concurrent CreateIndex. But message Rewrite in protos/transaction.proto has no field for it: the Rust→proto conversion matches and discards it, the proto→Rust
conversion hardcodes None. Since the resolver reads competing transactions back through that round trip, it can only ever see frag_reuse_index: None and take the fragment-overlap branch — so a deferred-remap compaction conflicts with a concurrent index build even though the exemption exists to prevent exactly that.
Not a wrong answer; wasted work. A matrix rejection isn't retried inside lance's commit loop, so the caller rebuilds the whole index.
Mechanism — the exemption is sound (eager remap fixes an enumerated list of indexes and can't fix one created concurrently, so it must conflict; deferred remap records an address-keyed translation that applies to any index, so it needn't). It's just unreachable, because the proto has no field and both conversions drop it.
Steps to reproduce
Two deterministic unit tests, no workload needed: (A) round-trip a Rewrite carrying an FRI and assert it survives — observed `before=true after=false`; (B) a 3-row table of `check_txn` verdicts showing overlap+None → `RetryableCommitConflict`, `no-overlap+None` → `Ok, and `overlap+Some` → `Ok`, i.e. the
exemption that never fires.
#[test]
fn test_rewrite_frag_reuse_index_roundtrips() {
// A Rewrite committed with deferred index remap carries the fragment-reuse index that
// the conflict resolver uses to exempt it from conflicting with a concurrent
// CreateIndex. Competing transactions are read back through this conversion, so if the
// field does not survive, the exemption can never apply.
let fri = IndexMetadata {
uuid: Uuid::new_v4(),
name: "__lance_frag_reuse".to_owned(),
fields: vec![],
covering_fields: vec![],
dataset_version: 7,
fragment_bitmap: Some(roaring::RoaringBitmap::from_iter([0u32, 1])),
index_details: None,
index_version: 0,
created_at: None,
base_id: None,
files: None,
};
let txn = Transaction::new(
7,
Operation::Rewrite {
groups: vec![RewriteGroup {
old_fragments: vec![Fragment::new(0)],
new_fragments: vec![Fragment::new(9)],
}],
rewritten_indices: vec![],
frag_reuse_index: Some(fri.clone()),
},
None,
);
let decoded = Transaction::try_from(pb::Transaction::from(&txn)).unwrap();
match decoded.operation {
Operation::Rewrite {
frag_reuse_index, ..
} => assert_eq!(
frag_reuse_index.map(|i| i.uuid),
Some(fri.uuid),
"the fragment-reuse index did not survive the round trip, so \
check_create_index_txn always sees None and takes the conflict branch"
),
other => panic!("expected Rewrite, got {other:?}"),
}
}
Expected behavior
Aforementioned transactions (CreateIndex and Rewrite) under the given conditions should not conflict.
Lance version
9.0, but I believe it's still present on tip of main
Language binding
Rust
Environment
No response
Logs / traceback
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Rewrite message in protos/transaction.proto and the Rust-to-proto and proto-to-Rust conversions described in the issue. Run or add the supplied round-trip test, then exercise the check_txn verdict matrix. Done means frag_reuse_index survives serialization and the deferred-remap Rewrite does not conflict with the concurrent CreateIndex case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100