lance-format / lance-format/lance

Refine "scalar index" concept

Open
#7,104 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-index enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

We have been using the term "scalar index" for a while but it has become both too general and too specific. For example:

  • FTS indexes are technically a "scalar index" but they are not usable in most places that scalar indexes are usable.
  • The term "scalar" is somewhat ambiguous now that we can index strings, structs, lists, and other things.
  • I'd like to eventually introduce sketches which accelerate aggregates but can't accelerate filters like other scalar indexes can.

Instead I think we just have "indexes" and they can provide one or more capabilities. Index being a very vague term that is more like "data plugins" or "alternative data representation". It's just supplementary data, which can be recomputed from the primary data, which accelerates some workflows.

We then have a number of extension points like:

  • Can provide vector search
  • Can provide full-text search
  • Can accelerate filters
  • Can accelerate aggregates
  • Etc.

The goal of this issue could be just documentation but could also be defining traits as well as other things.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inventorying the existing uses of “scalar index” and the current index extension points; no files or tests are identified in the issue. Clarify whether the result is documentation only or also capability traits for vector search, full-text search, filtering, and aggregates, then define completion as an agreed terminology and documented or implemented scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.