[DistDGL] Performance degradation on CPU in GraphSAGE
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
Performance degradation in `dgl/examples/pytorch/graphsage/dist` example on CPU after dataloader change [#4269](https://github.com/dmlc/dgl/pull/4269)
## To Reproduce
Steps to reproduce the behavior:
1. Prepare ogbn-papers100M dataset (8 part split):
```python3 partition_graph.py --dataset ogb-paper100M --num_parts 8 --output parts_8 --undirected --balance_train --balance_edges```
2. Run example
```
python3 ~/workspace/dgl/tools/launch.py --workspace ~/workspace/dgl/examples/pytorch/graphsage/dist/ --num_omp_threads 20 --num_trainers 1 --num_samplers 2 --num_servers 1 --part_config data/ogb-paper100M.json --ip_config ~/workspace/config.txt "python3 train_dist.py --graph_name ogb-paper100M --ip_config ~/workspace/config.txt --num_epochs 10 --eval_every 20 --num_hidden 256 --num
_layers 3 --fan_out 15,10,5 --lr 0.006"
```
Performance after:
```
Part 7, Epoch Time(s): 64.7051, sample+data_copy: 2.9623, forward: 9.5219, backward: 13.0456, update: 0.3492, #seeds: 150897, #inputs: 79792934
Part 3, Epoch Time(s): 64.7056, sample+data_copy: 3.0020, forward: 10.0696, backward: 25.9915, update: 0.3801, #seeds: 150897, #inputs: 78628489
Part 1, Epoch Time(s): 64.7045, sample+data_copy: 2.9547, forward: 8.8040, backward: 20.5609, update: 0.3706, #seeds: 150898, #inputs: 54821743
Part 0, Epoch Time(s): 64.7054, sample+data_copy: 3.0686, forward: 9.5668, backward: 13.2294, update: 0.3551, #seeds: 150898, #inputs: 75516969
Part 4, Epoch Time(s): 64.7055, sample+data_copy: 3.0451, forward: 9.5693, backward: 17.8947, update: 0.3636, #seeds: 150897, #inputs: 71458995
Part 5, Epoch Time(s): 64.7044, sample+data_copy: 3.0510, forward: 9.4090, backward: 13.5122, update: 0.3817, #seeds: 150897, #inputs: 71535302
Part 2, Epoch Time(s): 64.7055, sample+data_copy: 2.9399, forward: 9.4208, backward: 18.8902, update: 0.3657, #seeds: 150898, #inputs: 72412963
Part 6, Epoch Time(s): 64.7053, sample+data_copy: 3.0164, forward: 9.9258, backward: 21.2750, update: 0.3593, #seeds: 150897, #inputs: 66156315
```
## Expected behavior
Performance before:
```
Part 7, Epoch Time(s): 47.9451, sample+data_copy: 3.2357, forward: 16.4843, backward: 22.5896, update: 0.2777, #seeds: 150897, #inputs: 54925513
Part 3, Epoch Time(s): 47.9452, sample+data_copy: 3.1326, forward: 16.7308, backward: 22.3739, update: 0.2896, #seeds: 150897, #inputs: 56022318
Part 0, Epoch Time(s): 47.9452, sample+data_copy: 3.6224, forward: 18.1099, backward: 20.5638, update: 0.2884, #seeds: 150898, #inputs: 55908938
Part 1, Epoch Time(s): 47.9448, sample+data_copy: 3.2569, forward: 13.9563, backward: 26.9183, update: 0.3056, #seeds: 150898, #inputs: 39964856
Part 4, Epoch Time(s): 47.9451, sample+data_copy: 3.7290, forward: 17.6986, backward: 20.4828, update: 0.3175, #seeds: 150897, #inputs: 51627543
Part 2, Epoch Time(s): 47.9455, sample+data_copy: 3.2986, forward: 16.4955, backward: 22.5318, update: 0.2789, #seeds: 150898, #inputs: 52221848
Part 5, Epoch Time(s): 47.9445, sample+data_copy: 3.8029, forward: 16.2801, backward: 22.7546, update: 0.3441, #seeds: 150897, #inputs: 50720230
Part 6, Epoch Time(s): 47.9454, sample+data_copy: 3.7157, forward: 15.9476, backward: 23.2802, update: 0.3191, #seeds: 150897, #inputs: 46263629
```
## Environment
- DGL Version (e.g., 1.0): 1.1
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): PyTorch 1.13.1+cpu
- OS (e.g., Linux): Ubuntu 20.04
- How you installed DGL (`conda`, `pip`, source): pip
- Build command you used (if compiling from source):
```
git clone https://github.com/dmlc/dgl.git
cd dgl
git submodule update --init --recursive
mkdir build
cd build
cmake ..
make -j32
```
- Python version: Python 3.8.10
- CUDA/cuDNN version (if applicable): n/a
- GPU models and configuration (e.g. V100): CPU Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
- Any other relevant information:
8 x r5.16xlarge AWS instances
## Additional context
The issue doesn't reproduce with tcmalloc
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.