lance-format / lance-format/lance

Thread DataFusion MemoryPool through all index builds

Open
#7,301 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-index epic performance rust
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Give index builds a uniform, reactive memory model using DataFusion's existing MemoryPool (already a dependency, already used in merge_insert.rs) rather than each family's bespoke accounting.

A per-build FairSpillPool is sized from a memory budget. Builders grow/shrink a MemoryReservation; try_grow returning Err is the spill signal — the caller reacts by spilling intermediate state (the pool itself spills nothing). On a large machine with ample budget, the identical path stays fully in memory with no spill — no config change between small and large machines.

Reaction to the spill signal lands intermediate state in a SpillStore (#7300, and the threading work in #7302).

HNSW is out of scope — graph construction stays memory-heavy for now.

This is the umbrella issue. Per-family work:

  • FTS / inverted — #7304
  • Vector IVF — #7305
  • Scalar btree/bitmap — #7306

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing DataFusion MemoryPool usage in merge_insert.rs, then review the linked SpillStore and threading work in #7300 and #7302. Use the per-family issues #7304, #7305, and #7306 to identify the concrete index builders; completion means FTS, vector IVF, and scalar btree/bitmap builds share the reactive memory-reservation and spill-signal model, while HNSW remains out of scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.