conda-forge / conda-forge/python-spams-feedstock
Permanently locked to one thread even after specifying numThreads>1
- Dominant language
- Shell
- Stars
- 22
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
Issue:
When calling `spams.fistaFlat(..., numThreads, ...)` with `numThreads > 1` or `numThreads = -1` (all available cores/threads), two issues occur:
- Only one thread is actually used (verified by `htop`) even on a very large problem that uses all 6 cores with another implementation (e.g., numpy/numba).
- Any multiprocessing code that is run thereafter becomes locked to only one thread. For example, if I call `np.dot` before spams, then it can use all 6 cores. But if I call `np.dot` after calling `spams.fistaFlat`, then it only uses 1 core (verified by running things interactively in `jupyter` and monitoring `htop`; a kernel reset is required in order to "unlock" the cores again).
Possibly related to https://github.com/swig/swig/issues/927 or some issue where SWIG will change the state of the Python GIL? I have not yet checked if this issue occurs with the `python-mkl` package on pypi.
Environment (conda list):
```
# Name Version Build Channel
_libgcc_mutex 0.1 main
_tflow_select 2.1.0 gpu
absl-py 0.7.1 py37_0
appdirs 1.4.3 pypi_0 pypi
asn1crypto 0.24.0 pypi_0 pypi
astor 0.8.0 py37_0
atomicwrites 1.3.0 pypi_0 pypi
attrs 19.1.0 pypi_0 pypi
backcall 0.1.0 pypi_0 pypi
bcrypt 3.1.7 pypi_0 pypi
blas 1.0 mkl
bleach 3.1.0 pypi_0 pypi
c-ares 1.15.0 h7b6447c_1001
ca-certificates 2019.10.16 0
certifi 2019.9.11 py37_0
cffi 1.12.3 pypi_0 pypi
cryptography 2.7 pypi_0 pypi
cudatoolkit 10.0.130 0
cudnn 7.6.0 cuda10.0_0
cupti 10.0.130 0
cycler 0.10.0 pypi_0 pypi
decorator 4.4.0 pypi_0 pypi
defusedxml 0.6.0 pypi_0 pypi
entrypoints 0.3 pypi_0 pypi
flake8 3.7.8 pypi_0 pypi
gast 0.2.2 py37_0
google-pasta 0.1.7 py_0
grpcio 1.16.1 py37hf8bcb03_1
h5py 2.9.0 py37h7918eee_0
hdf5 1.10.4 hb1b8bf9_0
importlib-metadata 0.23 pypi_0 pypi
intel-openmp 2019.4 243
ipykernel 5.1.2 pypi_0 pypi
ipython 7.8.0 pypi_0 pypi
ipython-genutils 0.2.0 pypi_0 pypi
jedi 0.15.1 pypi_0 pypi
jinja2 2.10.1 pypi_0 pypi
joblib 0.13.2 pypi_0 pypi
jsonschema 3.0.2 pypi_0 pypi
jupyter-client 5.3.1 pypi_0 pypi
jupyter-core 4.5.0 pypi_0 pypi
keras-applications 1.0.8 py_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.1.0 pypi_0 pypi
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libprotobuf 3.8.0 hd408876_0
libstdcxx-ng 9.1.0 hdf63c60_0
llvmlite 0.30.0 pypi_0 pypi
mako 1.1.0 pypi_0 pypi
markdown 3.1.1 py37_0
markupsafe 1.1.1 pypi_0 pypi
matplotlib 3.1.1 pypi_0 pypi
mccabe 0.6.1 pypi_0 pypi
mistune 0.8.4 pypi_0 pypi
mkl 2019.4 243
mkl-service 2.3.0 py37he904b0f_0
mkl_fft 1.0.14 py37ha843d7b_0
mkl_random 1.0.2 py37hd81dba3_0
more-itertools 7.2.0 pypi_0 pypi
nbconvert 5.6.0 pypi_0 pypi
nbformat 4.4.0 pypi_0 pypi
ncurses 6.1 he6710b0_1
nengo 2.8.0 pypi_0 pypi
nengo-dl 3.0.0.dev0 dev_0
nengo-fpga 0.2.2.dev0 dev_0
nengo-ocl 1.4.0 pypi_0 pypi
nengolib 0.5.2 pypi_0 pypi
notebook 6.0.1 pypi_0 pypi
numba 0.46.0 pypi_0 pypi
numpy 1.16.5 py37h7e9f1db_0
numpy-base 1.16.5 py37hde5b4d6_0
opencv-python 4.1.1.26 pypi_0 pypi
openssl 1.1.1d h7b6447c_3
opt_einsum 3.1.0 py_0
packaging 19.2 pypi_0 pypi
pandas 0.25.1 pypi_0 pypi
pandocfilters 1.4.2 pypi_0 pypi
paramiko 2.6.0 pypi_0 pypi
parso 0.5.1 pypi_0 pypi
pexpect 4.7.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pip 19.2.2 py37_0
pluggy 0.13.0 pypi_0 pypi
progressbar2 3.43.1 pypi_0 pypi
prometheus-client 0.7.1 pypi_0 pypi
prompt-toolkit 2.0.9 pypi_0 pypi
protobuf 3.8.0 py37he6710b0_0
psutil 5.6.3 pypi_0 pypi
ptyprocess 0.6.0 pypi_0 pypi
py 1.8.0 pypi_0 pypi
pycodestyle 2.5.0 pypi_0 pypi
pycparser 2.19 pypi_0 pypi
pyflakes 2.1.1 pypi_0 pypi
pygments 2.4.2 pypi_0 pypi
pynacl 1.3.0 pypi_0 pypi
pynvml 8.0.2+8.gd435840 dev_0
pyopencl 2019.1.1 pypi_0 pypi
pyparsing 2.4.2 pypi_0 pypi
pyrsistent 0.15.4 pypi_0 pypi
pytest 5.2.1 pypi_0 pypi
python 3.7.4 h265db76_1
python-dateutil 2.8.0 pypi_0 pypi
python-utils 2.3.0 pypi_0 pypi
pytools 2019.1.1 pypi_0 pypi
pytz 2019.2 pypi_0 pypi
pyzmq 18.1.0 pypi_0 pypi
readline 7.0 h7b6447c_5
s-tui 1.0.0b2 dev_0
scikit-learn 0.21.3 pypi_0 pypi
scipy 1.3.1 py37h7c811a0_0
seaborn 0.9.0 pypi_0 pypi
send2trash 1.5.0 pypi_0 pypi
setuptools 41.0.1 py37_0
six 1.12.0 py37_0
spams-mkl 2.6.1 pypi_0 pypi
sqlite 3.29.0 h7b6447c_0
tensorboard 2.0.0 pyhb230dea_0
tensorflow 2.0.0 gpu_py37h768510d_0
tensorflow-base 2.0.0 gpu_py37h0ec5d1f_0
tensorflow-estimator 2.0.0 pyh2649769_0
tensorflow-gpu 2.0.0 h0d30ee6_0
termcolor 1.1.0 py37_1
terminado 0.8.2 pypi_0 pypi
testpath 0.4.2 pypi_0 pypi
tk 8.6.8 hbc83047_0
tornado 6.0.3 pypi_0 pypi
traitlets 4.3.2 pypi_0 pypi
urwid 2.0.1 pypi_0 pypi
wcwidth 0.1.7 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
werkzeug 0.15.5 py_0
wheel 0.33.4 py37_0
wrapt 1.11.2 py37h7b6447c_0
xz 5.2.4 h14c3975_4
zipp 0.6.0 pypi_0 pypi
zlib 1.2.11 h7b6447c_3
```
Details about conda and system ( conda info ):
```
active environment : ******
active env location : /home/arvoelke/anaconda3/envs/******
shell level : 2
user config file : /home/arvoelke/.condarc
populated config files : /home/arvoelke/.condarc
conda version : 4.7.12
conda-build version : 3.18.8
python version : 3.7.3.final.0
virtual packages : __cuda=10.1
base environment : /home/arvoelke/anaconda3 (writable)
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/arvoelke/anaconda3/pkgs
/home/arvoelke/.conda/pkgs
envs directories : /home/arvoelke/anaconda3/envs
/home/arvoelke/.conda/envs
platform : linux-64
user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.3 Linux/5.0.0-36-generic ubuntu/18.04.3 glibc/2.27
UID:GID : 1000:1000
netrc file : None
offline mode : False
```
Contributor guide
Assessment
This issue has not been assessed yet.