Segfault in smooth_knn_dist
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
I am getting a `Segmentation fault (core dumped)` on `fit` with any input data on linux.
UMAP version: `0.4.2` (happens with `0.4.1` as well)
OS: `centos-release-7-6.1810.2.el7.centos.x86_64`
I also tried running using Binder on ubuntu but there it works all OK.
Traceback with `python -q -X faulthandler`:
```
Current thread 0x00007f43ce617740 (most recent call first):
File "/SFS/user/wp/prihodad/git/usum-hpc/condaenv/lib/python3.7/site-packages/umap/umap_.py", line 607 in fuzzy_simplicial_set
File "/SFS/user/wp/prihodad/git/usum-hpc/condaenv/lib/python3.7/site-packages/umap/umap_.py", line 1777 in fit
File "/SFS/user/wp/prihodad/git/usum-hpc/condaenv/lib/python3.7/site-packages/umap/umap_.py", line 1973 in fit_transform
```
I added print statements before the `smooth_knn_dist` call and got:
```
local_connectivity 1.0
knn_dists [[0. 0.02700001 0.02700001 ... 0.03100002 0.03500003 0.03500003]
[0. 0. 0. ... 0.10399997 0.10399997 0.10399997]
[0. 0. 0. ... 0.10399997 0.10399997 0.10399997]
...
[0. 0.03799999 0.03799999 ... 0.04699999 0.04699999 0.04699999]
[0. 0.009 0.009 ... 0.07499999 0.08499998 0.09399998]
[0. 0.04699999 0.05699998 ... 0.12300003 0.13200003 0.14200002]]
n_neighbors 15
Fatal Python error: Segmentation fault
```
So the error happens somewhere in the `smooth_knn_dist` function.
Output of `pip freeze`:
```
biopython==1.76
bokeh==2.0.2
certifi==2020.4.5.1
chardet==3.0.4
click==7.1.2
cloudpickle==1.4.1
colorcet==2.0.2
cycler==0.10.0
dask==2.15.0
datashader==0.10.0
datashape==0.5.2
decorator==4.4.2
distributed==2.15.2
fsspec==0.7.3
HeapDict==1.0.1
holoviews==1.13.2
idna==2.9
imageio==2.8.0
Jinja2==2.11.2
joblib==0.14.1
kiwisolver==1.2.0
llvmlite==0.32.0
locket==0.2.0
Markdown==3.2.1
MarkupSafe==1.1.1
matplotlib==3.2.1
msgpack==1.0.0
multipledispatch==0.6.0
networkx==2.4
numba==0.49.0
numpy==1.18.4
packaging==20.3
pandas==1.0.3
panel==0.9.5
param==1.9.3
partd==1.1.0
Pillow==7.1.2
psutil==5.7.0
pyct==0.4.6
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
pyviz-comms==0.7.4
PyWavelets==1.1.1
PyYAML==5.3.1
requests==2.23.0
scikit-image==0.16.2
scikit-learn==0.22.2.post1
scipy==1.4.1
six==1.14.0
sortedcontainers==2.1.0
tbb==2020.0.133
tblib==1.6.0
toolz==0.10.0
tornado==6.0.4
tqdm==4.46.0
typing-extensions==3.7.4.2
umap-learn==0.4.2
urllib3==1.25.9
usum==0.1.5
xarray==0.15.1
```
Output of `conda env export`:
```
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=0_gnu
- ca-certificates=2020.4.5.1=hecc5488_0
- certifi=2020.4.5.1=py37hc8dfbb8_0
- ld_impl_linux-64=2.34=h53a641e_0
- libffi=3.2.1=he1b5a44_1007
- libgcc-ng=9.2.0=h24d8f2e_2
- libgomp=9.2.0=h24d8f2e_2
- libstdcxx-ng=9.2.0=hdf63c60_2
- ncurses=6.1=hf484d3e_1002
- openssl=1.1.1g=h516909a_0
- pip=20.1=pyh9f0ad1d_0
- python=3.7.6=h8356626_5_cpython
- python_abi=3.7=1_cp37m
- readline=8.0=hf8c457e_0
- setuptools=46.1.3=py37hc8dfbb8_0
- sqlite=3.30.1=hcee41ef_0
- tk=8.6.10=hed695b0_0
- wheel=0.34.2=py_1
- xz=5.2.5=h516909a_0
- zlib=1.2.11=h516909a_1006
- pip:
- biopython==1.76
- bokeh==2.0.2
- chardet==3.0.4
- click==7.1.2
- cloudpickle==1.4.1
- colorcet==2.0.2
- cycler==0.10.0
- dask==2.15.0
- datashader==0.10.0
- datashape==0.5.2
- decorator==4.4.2
- distributed==2.15.2
- fsspec==0.7.3
- heapdict==1.0.1
- holoviews==1.13.2
- idna==2.9
- imageio==2.8.0
- jinja2==2.11.2
- joblib==0.14.1
- kiwisolver==1.2.0
- llvmlite==0.32.0
- locket==0.2.0
- markdown==3.2.1
- markupsafe==1.1.1
- matplotlib==3.2.1
- msgpack==1.0.0
- multipledispatch==0.6.0
- networkx==2.4
- numba==0.49.0
- numpy==1.18.4
- packaging==20.3
- pandas==1.0.3
- panel==0.9.5
- param==1.9.3
- partd==1.1.0
- pillow==7.1.2
- psutil==5.7.0
- pyct==0.4.6
- pyparsing==2.4.7
- python-dateutil==2.8.1
- pytz==2020.1
- pyviz-comms==0.7.4
- pywavelets==1.1.1
- pyyaml==5.3.1
- requests==2.23.0
- scikit-image==0.16.2
- scikit-learn==0.22.2.post1
- scipy==1.4.1
- six==1.14.0
- sortedcontainers==2.1.0
- tbb==2020.0.133
- tblib==1.6.0
- toolz==0.10.0
- tornado==6.0.4
- tqdm==4.46.0
- typing-extensions==3.7.4.2
- umap-learn==0.4.2
- urllib3==1.25.9
- usum==0.1.5
- xarray==0.15.1
```
Contributor guide
Research direction
Start in umap/umap_.py at fuzzy_simplicial_set and the smooth_knn_dist call identified by the traceback. Reproduce the failure with the reported Python 3.7, UMAP 0.4.x, Linux, and dependency versions, then isolate the Linux-specific cause. Done means fit completes without a segmentation fault on the reported environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100