[DistDGL] GraphSAGE example crashes on ogbn-papers100M dataset
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
`dgl/examples/pytorch/graphsage/dist` example crashes after [#4269](https://github.com/dmlc/dgl/pull/4269)
## To Reproduce
Steps to reproduce the behavior:
1. Prepare ogbn-papers100M dataset (8 part split):
```python3 partition_graph.py --dataset ogb-paper100M --num_parts 8 --output parts_8 --undirected --balance_train --balance_edges```
2. Run example
```
python3 ~/workspace/dgl/tools/launch.py --workspace ~/workspace/dgl/examples/pytorch/graphsage/dist/ --num_omp_threads 20 --num_trainers 1 --num_samplers 2 --num_servers 1 --part_config data/ogb-paper100M.json --ip_config ~/workspace/config.txt "python3 train_dist.py --graph_name ogb-paper100M --ip_config ~/workspace/config.txt --num_epochs 10 --eval_every 20 --num_hidden 256 --num
_layers 3 --fan_out 15,10,5 --lr 0.006"
```
Error message:
```
part 5, train: 150897 (local: 127740), val: 15658 (local: 0), test: 26792 (local: 0)
part 6, train: 150897 (local: 144545), val: 15658 (local: 0), test: 26792 (local: 0)
#labels: 172
#labels: 172
#labels: 172
#labels: 172
#labels: 172
Traceback (most recent call last):
File "train_dist.py", line 440, in
main(args)
File "train_dist.py", line 379, in main
run(args, device, data)
File "train_dist.py", line 208, in run
batch_inputs, batch_labels = load_subtensor(
File "train_dist.py", line 20, in load_subtensor
g.ndata["features"][input_nodes].to(device) if load_feat else None
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/distributed/dist_tensor.py", line 205, in __getitem__
return self.kvstore.pull(name=self._name, id_tensor=idx)
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/distributed/kvstore.py", line 1463, in pull
return rpc.fast_pull(
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/distributed/rpc.py", line 1166, in fast_pull
res_tensor = _CAPI_DGLRPCFastPull(
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/_ffi/_ctypes/function.py", line 212, in __call__
check_call(
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/_ffi/base.py", line 70, in check_call
raise DGLError(py_str(_LIB.DGLGetLastError()))
dgl._ffi.base.DGLError: [13:01:20] /home/ubuntu/dgl_old/dgl/src/rpc/rpc.cc:489: Check failed: p_id < machine_count (8 vs. 8) : Invalid partition ID.
Stack trace:
[bt] (0) /usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/libdgl.so(+0x5ec8b8) [0x7f1c369d78b8]
[bt] (1) /usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/libdgl.so(+0x5efe0b) [0x7f1c369dae0b]
[bt] (2) /usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/libdgl.so(+0x5f1158) [0x7f1c369dc158]
[bt] (3) /usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/libdgl.so(DGLFuncCall+0x60) [0x7f1c36832760]
[bt] (4) /lib/x86_64-linux-gnu/libffi.so.7(+0x6ff5) [0x7f1c52c47ff5]
[bt] (5) /lib/x86_64-linux-gnu/libffi.so.7(+0x640a) [0x7f1c52c4740a]
[bt] (6) /usr/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(_ctypes_callproc+0x5b6) [0x7f1c52c60306]
[bt] (7) /usr/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x139dc) [0x7f1c52c609dc]
[bt] (8) /usr/bin/python3(_PyObject_MakeTpCall+0x296) [0x5f7056]
```
## Expected behavior
No crash
## Environment
- DGL Version (e.g., 1.0): 1.1
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): PyTorch 1.13.1+cpu
- OS (e.g., Linux): Ubuntu 20.04
- How you installed DGL (`conda`, `pip`, source): pip
- Build command you used (if compiling from source):
```
git clone https://github.com/dmlc/dgl.git
cd dgl
git submodule update --init --recursive
mkdir build
cd build
cmake ..
make -j32
```
- Python version: Python 3.8.10
- CUDA/cuDNN version (if applicable): n/a
- GPU models and configuration (e.g. V100): CPU Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
- Any other relevant information:
8 x r5.16xlarge AWS instances
## Additional context
The issue doesn't reproduce with tcmalloc
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.