lance-format / lance-format/lance
Refine "scalar index" concept
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
We have been using the term "scalar index" for a while but it has become both too general and too specific. For example:
- FTS indexes are technically a "scalar index" but they are not usable in most places that scalar indexes are usable.
- The term "scalar" is somewhat ambiguous now that we can index strings, structs, lists, and other things.
- I'd like to eventually introduce sketches which accelerate aggregates but can't accelerate filters like other scalar indexes can.
Instead I think we just have "indexes" and they can provide one or more capabilities. Index being a very vague term that is more like "data plugins" or "alternative data representation". It's just supplementary data, which can be recomputed from the primary data, which accelerates some workflows.
We then have a number of extension points like:
- Can provide vector search
- Can provide full-text search
- Can accelerate filters
- Can accelerate aggregates
- Etc.
The goal of this issue could be just documentation but could also be defining traits as well as other things.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inventorying the existing uses of “scalar index” and the current index extension points; no files or tests are identified in the issue. Clarify whether the result is documentation only or also capability traits for vector search, full-text search, filtering, and aggregates, then define completion as an agreed terminology and documented or implemented scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, search
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100