lmcinnes / lmcinnes/pynndescent

Segfault with n_jobs > 17 during indexing

Open
#142 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
970
Forks
107
PR merge metrics
No merged PRs in 30d

Description

Hi,
I ran into this weird bug: pynndescent works fine when using a smaller number of n_jobs (anything up to n_jobs=17 works) , but above, it just throws a segmentation fault. Note that I'm running this on a 32-CPU server, so n_jobs=18 is nowhere near the physical number of CPUs.

from pynndescent import NNDescent
import numpy as np
x =np.random.randn(10_000, 20)

index = NNDescent(x, verbose=True, n_jobs=16) # works fine
index = NNDescent(x, verbose=True, n_jobs=17) # works fine
index = NNDescent(x, verbose=True, n_jobs=18) # --> segfault

# Thu Sep 23 03:05:20 2021 Building RP forest with 23 trees
# Thu Sep 23 03:05:21 2021 NN descent for 13 iterations
# 	 1  /  17
# [1]    2730 segmentation fault  ipython

The size of the input data doesn't seem to matter much, I get the same breakdown at n_jobs=18 with 100k instead of 10k datapoints (or 200 instead of 20 features).


Package Versions

python                    3.7.7                hcff3b4d_5  
numba                     0.52.0           py37hdc94413_0    conda-forge
pynndescent               0.5.4              pyh6c4a22f_0    conda-forge

EDIT: This seems to be specific to the conda-forge version, cloning the current commit and doing a pip install -e . allows the above code to run without the segfault. No idea what the problem is though.

Even weirder: Cloning, checking out 0.5.4 (which is the commit the conda-forge version is based on) and doing a pip install -e . also DOES NOT produce the seqfault. Must be something in that conda-forge package...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the crash with the conda-forge versions listed in the issue using the provided NNDescent example and n_jobs=18. Compare that environment with editable installs from commit 4047550 and tag 0.5.4, then investigate the packaging difference. Done means the cause of the conda-forge-only segmentation fault is identified and the failing configuration no longer crashes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.