lance-format / lance-format/lance
Index reads fail with `Not found: .../tree/<parent>/_indices/...` through a branch created from another branch
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
An index created on a dataset is reachable through a first-generation branch (shallow clone) via the clone's base_paths redirect, but a branch created from that branch records its index redirect against the immediate parent tree instead of composing the parent's own redirect to where the index files physically live. Every index-consuming read through the second-generation branch then fails with a Not found on .../tree/<parent>/_indices/....
resolve_base_uri resolves a file's location by looking up its base_id in manifest.base_paths. A shallow clone (Operation::Clone from Dataset::create_branch) populates base_paths with a redirect to the source tree. When the source is itself a clone, the redirect points at the source's tree path (.../tree/<parent>/_indices/...) — but the index files were never copied there; the parent reached them through its own base_paths entry, which the grandchild does not chain to.
Repro (Rust, v9.0.0-beta.21 / rev 1aec1465; both IndexType::BTree and IndexType::Inverted reproduce it):
// dataset + a scalar index on `value`
let mut ds = /* write small dataset */;
ds.create_index_builder(&["value"], IndexType::BTree, &ScalarIndexParams::default())
.replace(true).await?;
// first-generation branch + a write — indexed reads WORK here
let mut feature = ds.create_branch("feature", ds.version().version, None).await?;
feature.append(/* one batch */, None).await?;
scan_with_filter(&feature, "value = 1").await?; // ok
// second-generation branch: clone the clone
let experiment = feature
.create_branch("experiment", feature.version().version, None).await?;
// the same indexed read now fails:
// Not found: .../tree/feature/_indices/<uuid>/page_lookup.lance
scan_with_filter(&experiment, "value = 1").await; // Err(NotFound)
BTREE surfaces it as a missing page_lookup.lance, inverted/FTS as a missing tokens.lance. The files exist under the original dataset root's _indices/. A non-indexed scan of the same second-generation branch returns the rows correctly, so this is specifically index-file resolution, not data-file resolution.
Expected: index reads through any depth of shallow-clone lineage resolve to the tree that physically holds the index files — either by composing the transitive base_paths redirect at clone time, or by resolving base_id transitively at read time.
Suggested validation: an indexed dataset, a branch-of-a-branch, and an indexed read through the grandchild that returns the same rows a non-indexed scan does; cover BTREE and inverted indexes, and a third-generation clone.
Related (checked; none is this bug): #4863 "shallow_clone support indexes" (closed) added index support for a single clone level — this repro shows one level works and the second level breaks. #7785 (use original base paths when writing to a branch) and #7515 (in-place materialize of base-referenced files) touch the same base_paths subsystem from the data-file / feature side. #4853 is the shallow_clone→branch epic. This is the index-file correctness face of that cluster and is not tracked by any of them.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with resolve_base_uri in rust/lance/src/dataset/files.rs and Operation::Clone in rust/lance/src/dataset.rs, then reproduce the Rust example with a branch-of-a-branch. Trace how base_paths is populated and resolved for index files. Done means indexed reads through second- and third-generation branches return the same rows as non-indexed scans, covering BTree and inverted indexes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100