lance-format / lance-format/lance
PQ assignment stage `HashMap` lookups are too slow at 1B rows
Open
Nobody has claimed this yet.
A-index
performance
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
We don't need a HashMap anyways. Since we're mapping from row address to partition id we can use a Vec<Vec<...>> where each lookup is map[fragment_id][row_offset]. From some experimentation this is ~6x faster.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the PQ assignment stage and inspect how row addresses are mapped to partition IDs. Compare the current HashMap lookup with the proposed nested-vector indexing, then benchmark the assignment path at large row counts. Done means the lookup change is validated without changing partition assignments and the reported performance improvement is reproduced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100