dmlc / dmlc/dgl

[DistDGL] Performance degradation on CPU in GraphSAGE

Open
#5,529 12 comments 0 reactions 1 assignee Claimed by @Rhett-Ying View on GitHub
bug:unconfirmed
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

Performance degradation in `dgl/examples/pytorch/graphsage/dist` example on CPU after dataloader change [#4269](https://github.com/dmlc/dgl/pull/4269)

## To Reproduce

Steps to reproduce the behavior:

1. Prepare ogbn-papers100M dataset (8 part split):
```python3 partition_graph.py --dataset ogb-paper100M --num_parts 8 --output parts_8 --undirected --balance_train --balance_edges```

2. Run example
```
python3 ~/workspace/dgl/tools/launch.py --workspace ~/workspace/dgl/examples/pytorch/graphsage/dist/ --num_omp_threads 20 --num_trainers 1 --num_samplers 2 --num_servers 1 --part_config data/ogb-paper100M.json --ip_config ~/workspace/config.txt "python3 train_dist.py --graph_name ogb-paper100M --ip_config ~/workspace/config.txt --num_epochs 10 --eval_every 20 --num_hidden 256 --num
_layers 3 --fan_out 15,10,5 --lr 0.006"
```

Performance after:
```
Part 7, Epoch Time(s): 64.7051, sample+data_copy: 2.9623, forward: 9.5219, backward: 13.0456, update: 0.3492, #seeds: 150897, #inputs: 79792934
Part 3, Epoch Time(s): 64.7056, sample+data_copy: 3.0020, forward: 10.0696, backward: 25.9915, update: 0.3801, #seeds: 150897, #inputs: 78628489
Part 1, Epoch Time(s): 64.7045, sample+data_copy: 2.9547, forward: 8.8040, backward: 20.5609, update: 0.3706, #seeds: 150898, #inputs: 54821743
Part 0, Epoch Time(s): 64.7054, sample+data_copy: 3.0686, forward: 9.5668, backward: 13.2294, update: 0.3551, #seeds: 150898, #inputs: 75516969
Part 4, Epoch Time(s): 64.7055, sample+data_copy: 3.0451, forward: 9.5693, backward: 17.8947, update: 0.3636, #seeds: 150897, #inputs: 71458995
Part 5, Epoch Time(s): 64.7044, sample+data_copy: 3.0510, forward: 9.4090, backward: 13.5122, update: 0.3817, #seeds: 150897, #inputs: 71535302
Part 2, Epoch Time(s): 64.7055, sample+data_copy: 2.9399, forward: 9.4208, backward: 18.8902, update: 0.3657, #seeds: 150898, #inputs: 72412963
Part 6, Epoch Time(s): 64.7053, sample+data_copy: 3.0164, forward: 9.9258, backward: 21.2750, update: 0.3593, #seeds: 150897, #inputs: 66156315
```

## Expected behavior

Performance before:
```
Part 7, Epoch Time(s): 47.9451, sample+data_copy: 3.2357, forward: 16.4843, backward: 22.5896, update: 0.2777, #seeds: 150897, #inputs: 54925513
Part 3, Epoch Time(s): 47.9452, sample+data_copy: 3.1326, forward: 16.7308, backward: 22.3739, update: 0.2896, #seeds: 150897, #inputs: 56022318
Part 0, Epoch Time(s): 47.9452, sample+data_copy: 3.6224, forward: 18.1099, backward: 20.5638, update: 0.2884, #seeds: 150898, #inputs: 55908938
Part 1, Epoch Time(s): 47.9448, sample+data_copy: 3.2569, forward: 13.9563, backward: 26.9183, update: 0.3056, #seeds: 150898, #inputs: 39964856
Part 4, Epoch Time(s): 47.9451, sample+data_copy: 3.7290, forward: 17.6986, backward: 20.4828, update: 0.3175, #seeds: 150897, #inputs: 51627543
Part 2, Epoch Time(s): 47.9455, sample+data_copy: 3.2986, forward: 16.4955, backward: 22.5318, update: 0.2789, #seeds: 150898, #inputs: 52221848
Part 5, Epoch Time(s): 47.9445, sample+data_copy: 3.8029, forward: 16.2801, backward: 22.7546, update: 0.3441, #seeds: 150897, #inputs: 50720230
Part 6, Epoch Time(s): 47.9454, sample+data_copy: 3.7157, forward: 15.9476, backward: 23.2802, update: 0.3191, #seeds: 150897, #inputs: 46263629
```

## Environment

- DGL Version (e.g., 1.0): 1.1
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): PyTorch 1.13.1+cpu
- OS (e.g., Linux): Ubuntu 20.04
- How you installed DGL (`conda`, `pip`, source): pip
- Build command you used (if compiling from source):
```
git clone https://github.com/dmlc/dgl.git
cd dgl
git submodule update --init --recursive
mkdir build
cd build
cmake ..
make -j32
```
- Python version: Python 3.8.10
- CUDA/cuDNN version (if applicable): n/a
- GPU models and configuration (e.g. V100): CPU Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
- Any other relevant information:
8 x r5.16xlarge AWS instances

## Additional context

The issue doesn't reproduce with tcmalloc

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.