lance-format / lance-format/lance
feat: make load_segment_params generic over scalar index types, not inverted-only
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Summary
lance::index::scalar::load_segment_params is the supported way to recover an existing index's build parameters from a committed segment, but it is implemented only for inverted indexes (rust/lance/src/index/scalar/inverted.rs, re-exported from rust/lance/src/index/scalar.rs). There is no equivalent for the other scalar index types, even though the underlying capability is already generic.
Why it's already generic
derive_index_params() is a method on the ScalarIndex trait and is implemented across the scalar plugins — including JsonIndex, whose implementation reconstructs the full JsonIndexParameters (path, target index type, target params, and target_data_type read back via target_index.training_data_type()).
So "open the scalar index for this segment, call derive_index_params()" would work for any scalar type. Only the inverted-specific wrapper exists.
Why it matters
A system that rebuilds indexes as maintenance needs the original build configuration, and a full rebuild via CreateIndexBuilder::replace(true) takes its configuration entirely from caller-supplied params — it does not read the existing index. So the caller must recover those params itself or the rebuild silently reverts to defaults.
For inverted indexes load_segment_params makes that possible. For every other scalar type the caller's options today are:
- read the manifest details, which for some types cannot round-trip the full configuration (see #9256 for the JSON case, and #4628 for the target index version), or
- keep a private copy of the configuration outside Lance, which then drifts from what Lance actually built.
A generic entry point would remove both.
Request
Promote load_segment_params (and, where useful, load_segment_details) to a type-agnostic function over scalar indexes, backed by derive_index_params(), rather than one that lives in the inverted module.
Related: #9256, #4628, #5311.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance/src/index/scalar/inverted.rs and rust/lance/src/index/scalar.rs, tracing load_segment_params, load_segment_details, and the ScalarIndex::derive_index_params() implementations. Make the supported loading entry point type-agnostic while preserving parameter recovery for inverted and other scalar indexes, including JsonIndex; completion should provide the generic API described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100