lance-format / lance-format/lance

Index reads fail with `Not found: .../tree/<parent>/_indices/...` through a branch created from another branch

Open
#7,840 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

An index created on a dataset is reachable through a first-generation branch (shallow clone) via the clone's base_paths redirect, but a branch created from that branch records its index redirect against the immediate parent tree instead of composing the parent's own redirect to where the index files physically live. Every index-consuming read through the second-generation branch then fails with a Not found on .../tree/<parent>/_indices/....

resolve_base_uri resolves a file's location by looking up its base_id in manifest.base_paths. A shallow clone (Operation::Clone from Dataset::create_branch) populates base_paths with a redirect to the source tree. When the source is itself a clone, the redirect points at the source's tree path (.../tree/<parent>/_indices/...) — but the index files were never copied there; the parent reached them through its own base_paths entry, which the grandchild does not chain to.

Repro (Rust, v9.0.0-beta.21 / rev 1aec1465; both IndexType::BTree and IndexType::Inverted reproduce it):

// dataset + a scalar index on `value`
let mut ds = /* write small dataset */;
ds.create_index_builder(&["value"], IndexType::BTree, &ScalarIndexParams::default())
    .replace(true).await?;

// first-generation branch + a write — indexed reads WORK here
let mut feature = ds.create_branch("feature", ds.version().version, None).await?;
feature.append(/* one batch */, None).await?;
scan_with_filter(&feature, "value = 1").await?;   // ok

// second-generation branch: clone the clone
let experiment = feature
    .create_branch("experiment", feature.version().version, None).await?;

// the same indexed read now fails:
//   Not found: .../tree/feature/_indices/<uuid>/page_lookup.lance
scan_with_filter(&experiment, "value = 1").await;   // Err(NotFound)

BTREE surfaces it as a missing page_lookup.lance, inverted/FTS as a missing tokens.lance. The files exist under the original dataset root's _indices/. A non-indexed scan of the same second-generation branch returns the rows correctly, so this is specifically index-file resolution, not data-file resolution.

Expected: index reads through any depth of shallow-clone lineage resolve to the tree that physically holds the index files — either by composing the transitive base_paths redirect at clone time, or by resolving base_id transitively at read time.

Suggested validation: an indexed dataset, a branch-of-a-branch, and an indexed read through the grandchild that returns the same rows a non-indexed scan does; cover BTREE and inverted indexes, and a third-generation clone.

Related (checked; none is this bug): #4863 "shallow_clone support indexes" (closed) added index support for a single clone level — this repro shows one level works and the second level breaks. #7785 (use original base paths when writing to a branch) and #7515 (in-place materialize of base-referenced files) touch the same base_paths subsystem from the data-file / feature side. #4853 is the shallow_clone→branch epic. This is the index-file correctness face of that cluster and is not tracked by any of them.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with resolve_base_uri in rust/lance/src/dataset/files.rs and Operation::Clone in rust/lance/src/dataset.rs, then reproduce the Rust example with a branch-of-a-branch. Trace how base_paths is populated and resolved for index files. Done means indexed reads through second- and third-generation branches return the same rows as non-indexed scans, covering BTree and inverted indexes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.