lmcinnes / lmcinnes/pynndescent
Very high memory usage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 970
- Forks
- 107
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I am experimenting excessive memory utilization of the library:
import numpy as np
X = np.random.rand(20_000_000, 2) # 20MM 2D vectors
index = pynndescent.NNDescent(X)
index.prepare()
The following code exceeds 30GB of memory in my machine. Surprisingly, other libraries like hnswlib are capable of building the same index in approximately 5GB of memory.
Is this a possible bug/memory leak in the library or is it the expected memory usage?
Testing with version 0.5.10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported Python snippet with pynndescent 0.5.10, focusing on memory use during NNDescent(X) and index.prepare() for the 20-million-vector input. Compare the observed usage with the report and determine whether it is expected behavior or a memory leak; done means the cause and expected usage are established, or a concrete bug is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100