lance-format / lance-format/lance

feat: make load_segment_params generic over scalar index types, not inverted-only

Open
#9,258 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Summary

lance::index::scalar::load_segment_params is the supported way to recover an existing index's build parameters from a committed segment, but it is implemented only for inverted indexes (rust/lance/src/index/scalar/inverted.rs, re-exported from rust/lance/src/index/scalar.rs). There is no equivalent for the other scalar index types, even though the underlying capability is already generic.

Why it's already generic

derive_index_params() is a method on the ScalarIndex trait and is implemented across the scalar plugins — including JsonIndex, whose implementation reconstructs the full JsonIndexParameters (path, target index type, target params, and target_data_type read back via target_index.training_data_type()).

So "open the scalar index for this segment, call derive_index_params()" would work for any scalar type. Only the inverted-specific wrapper exists.

Why it matters

A system that rebuilds indexes as maintenance needs the original build configuration, and a full rebuild via CreateIndexBuilder::replace(true) takes its configuration entirely from caller-supplied params — it does not read the existing index. So the caller must recover those params itself or the rebuild silently reverts to defaults.

For inverted indexes load_segment_params makes that possible. For every other scalar type the caller's options today are:

  • read the manifest details, which for some types cannot round-trip the full configuration (see #9256 for the JSON case, and #4628 for the target index version), or
  • keep a private copy of the configuration outside Lance, which then drifts from what Lance actually built.

A generic entry point would remove both.

Request

Promote load_segment_params (and, where useful, load_segment_details) to a type-agnostic function over scalar indexes, backed by derive_index_params(), rather than one that lives in the inverted module.

Related: #9256, #4628, #5311.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance/src/index/scalar/inverted.rs and rust/lance/src/index/scalar.rs, tracing load_segment_params, load_segment_details, and the ScalarIndex::derive_index_params() implementations. Make the supported loading entry point type-agnostic while preserving parameter recovery for inverted and other scalar indexes, including JsonIndex; completion should provide the generic API described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.