[FEA] Improve scale-up capabilities for dynamic indexes
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 854
- Forks
- 236
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 62
Description
Transactional workloads that need immediate consistency often sacrifice partition tolerance, requiring very large indexes to be built instead of breaking them into multiple shards.
As a result, this poses a problem to limited memory conditions (both device and host). We want to see how SCADA performs when used as a tiered memory access layer to enable spilling these indexes to disk. SCADA could be especially useful for graph-based methods like CAGRA and HNSW, since their construction and search require a lot of sparse random accesses, which can be done in massive parallel to saturate the disk (in theory).
This is a placeholder issue for a proof of concept to see if SCADA can improve index build (and search) perf on the GPU w/ graph-based methods for dynamic indexing scenarios.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by investigating how SCADA could be evaluated with the CAGRA and HNSW graph methods for tiered memory access, then define a proof of concept that compares GPU index-build and search performance when indexes spill to disk under limited-memory conditions.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, performance, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100