lance-format / lance-format/lance
Thread DataFusion MemoryPool through all index builds
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Give index builds a uniform, reactive memory model using DataFusion's existing MemoryPool (already a dependency, already used in merge_insert.rs) rather than each family's bespoke accounting.
A per-build FairSpillPool is sized from a memory budget. Builders grow/shrink a MemoryReservation; try_grow returning Err is the spill signal — the caller reacts by spilling intermediate state (the pool itself spills nothing). On a large machine with ample budget, the identical path stays fully in memory with no spill — no config change between small and large machines.
Reaction to the spill signal lands intermediate state in a SpillStore (#7300, and the threading work in #7302).
HNSW is out of scope — graph construction stays memory-heavy for now.
This is the umbrella issue. Per-family work:
- FTS / inverted — #7304
- Vector IVF — #7305
- Scalar btree/bitmap — #7306
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing DataFusion MemoryPool usage in merge_insert.rs, then review the linked SpillStore and threading work in #7300 and #7302. Use the per-family issues #7304, #7305, and #7306 to identify the concrete index builders; completion means FTS, vector IVF, and scalar btree/bitmap builds share the reactive memory-reservation and spill-signal model, while HNSW remains out of scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100