scikit-learn / scikit-learn/scikit-learn
Importing sklearn messes with CPU affinity, causing n_jobs=-1 to become useless
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 67.3k
- Forks
- 27.4k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 58
Description
Description
Please see a PoC notebook.
tl:dr: import sklearn causes CPU affinity to be set to {0}. Resetting it causes RandomizedSearchCV to freeze.
Expected Results
n_jobs=-1 uses all cores and does not freeze.
Actual Results
Only one core is loaded at 100% and nothing happens for 30+ minutes. Not resetting CPU affinity or not specifying n_jobs=-1 causes the freeze to go away.
Versions
Linux-4.14.36-1-MANJARO-x86_64-with-arch-Manjaro-Linux
Python 3.6.5 (default, Apr 12 2018, 22:45:43)
[GCC 7.3.1 20180312]
NumPy 1.14.2
SciPy 1.0.1
Scikit-Learn 0.19.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked PoC notebook and reproduce the issue using RandomizedSearchCV with n_jobs=-1 on the reported Linux and Python setup. Trace the import-time CPU-affinity change and the resulting parallel execution behavior; done means importing sklearn no longer leaves only one core available and the search completes while using all cores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100