CPU omp thread affinity becomes imbalanced after backpass
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
I am running graphsage app in dgl+pytorch setting. In minigun code which spawns omp threads to execute e.g copyReduce, I see that multiple omp threads are getting executed on a cpu - somehow thread affinity is getting messed up. This happens only after first backpass which is creating extra threads in addition to the thread created in forward pass. For example, if I set OMP_NUM_THREADS=10 (for 10 cpus), then after the first fwd and bck pass I see there are 20 os threads. Then in the next fwd pass a subset of 10 threads out of the pool of 20 threads are used, leading to the affinity problem.
## Environment
- DGL Version (e.g., 1.0): latest commit
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): pytorch
- OS (e.g., Linux): Centos
- How you installed DGL (`conda`, `pip`, source): compiled and installed from src code
- Build command you used (if compiling from source): make -j
- Python version: 3.6/3.7
- CUDA/cuDNN version (if applicable):
- GPU models and configuration (e.g. V100):
- Any other relevant information: Intel CPUs
## Additional context
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.