How could SPANN load DEEP-1B (358GB) into memory with only 128GB RAM (in SPANN paper)?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 5k
- Forks
- 622
- Avg merge
- 8h 37m
- Merged PRs (30d)
- 5
Description
Hi, I'm reading the source code and paper of SPANN.
In /AnnService/src/IndexBuilder/main.cpp, we can see data will be loaded in function DefaultVectorReader::GetVectorSet().
In that function, vectors are loaded in one single ReadBinary() method.
But if the size of origin file exceeds the RAM, how could do that? In SPANN paper, only uses 128GB RAM for DEEP-1B, which has 358GB basepoints file according to big-ann-benchmark.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with AnnService/src/IndexBuilder/main.cpp and trace DefaultVectorReader::GetVectorSet() into the single ReadBinary() call. Compare that loading path with the SPANN paper and the reported DEEP-1B dataset size; done means documenting how the 128GB-memory result is reconciled with the 358GB basepoints file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- search
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100