lmcinnes / lmcinnes/pynndescent
Memory corruption when using alternative algorithm
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 970
- Forks
- 107
- PR merge metrics
- No merged PRs in 30d
Description
In some cases (I haven't been able to find a pattern for this), the library can fail with memory corruption. I am able to get this consistently with the following code:
```python
>>> import numpy as np
>>> from pynndescent import NNDescent
>>> x = np.genfromtxt("mouse_sample_1500.txt", delimiter=",")
>>> index = NNDescent(x, algorithm="alternative")
Works fine!
>>> x = np.genfromtxt("mouse_sample_100.txt", delimiter=",")
>>> index = NNDescent(x, algorithm="alternative")
Works fine!
>>> x = np.genfromtxt("mouse_sample_1000.txt", delimiter=",")
>>> index = NNDescent(x, algorithm="alternative")
double free or corruption (out)
abort (core dumped) python
```
The data in question is a small dense (1000, 50) matrix. Bizarrely, a smaller (100, 50) and a larger (1500, 50) matrix work perfectly fine. I can consistently replicate this with the files attached below.
[mouse_sample_100.txt](https://github.com/lmcinnes/pynndescent/files/2659799/mouse_sample_100.txt)
[mouse_sample_1000.txt](https://github.com/lmcinnes/pynndescent/files/2659798/mouse_sample_1000.txt)
[mouse_sample_1500.txt](https://github.com/lmcinnes/pynndescent/files/2659781/mouse_sample_1500.txt)
I created an empty conda environment with `python=3.6.7`. I installed `numpy` and `pynndescent` using `pip`:
```
> pip freeze
certifi==2018.10.15
llvmlite==0.26.0
numba==0.41.0
numpy==1.15.4
pynndescent==0.2.1
scikit-learn==0.20.1
scipy==1.1.0
```
This does not occur using `algorithm="standard"`.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported NNDescent(x, algorithm="alternative") calls with the attached mouse_sample_100.txt, mouse_sample_1000.txt, and mouse_sample_1500.txt data, using the listed Python, NumPy, Numba, and pynndescent versions. Compare the alternative and standard algorithms and isolate why the 1000-by-50 case corrupts memory; done means the reproduction no longer crashes and the regression is covered by a test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100