dmlc / dmlc/dgl

[DistDGL] GraphSAGE example crashes on ogbn-papers100M dataset

Open
#5,528 17 comments 0 reactions 2 assignees Claimed by @TristonC View on GitHub
bug:confirmed
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

`dgl/examples/pytorch/graphsage/dist` example crashes after [#4269](https://github.com/dmlc/dgl/pull/4269)

## To Reproduce

Steps to reproduce the behavior:

1. Prepare ogbn-papers100M dataset (8 part split):
```python3 partition_graph.py --dataset ogb-paper100M --num_parts 8 --output parts_8 --undirected --balance_train --balance_edges```

2. Run example
```
python3 ~/workspace/dgl/tools/launch.py --workspace ~/workspace/dgl/examples/pytorch/graphsage/dist/ --num_omp_threads 20 --num_trainers 1 --num_samplers 2 --num_servers 1 --part_config data/ogb-paper100M.json --ip_config ~/workspace/config.txt "python3 train_dist.py --graph_name ogb-paper100M --ip_config ~/workspace/config.txt --num_epochs 10 --eval_every 20 --num_hidden 256 --num
_layers 3 --fan_out 15,10,5 --lr 0.006"
```

Error message:

```
part 5, train: 150897 (local: 127740), val: 15658 (local: 0), test: 26792 (local: 0)
part 6, train: 150897 (local: 144545), val: 15658 (local: 0), test: 26792 (local: 0)
#labels: 172
#labels: 172
#labels: 172
#labels: 172
#labels: 172
Traceback (most recent call last):
File "train_dist.py", line 440, in
main(args)
File "train_dist.py", line 379, in main
run(args, device, data)
File "train_dist.py", line 208, in run
batch_inputs, batch_labels = load_subtensor(
File "train_dist.py", line 20, in load_subtensor
g.ndata["features"][input_nodes].to(device) if load_feat else None
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/distributed/dist_tensor.py", line 205, in __getitem__
return self.kvstore.pull(name=self._name, id_tensor=idx)
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/distributed/kvstore.py", line 1463, in pull
return rpc.fast_pull(
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/distributed/rpc.py", line 1166, in fast_pull
res_tensor = _CAPI_DGLRPCFastPull(
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/_ffi/_ctypes/function.py", line 212, in __call__
check_call(
File "/usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/_ffi/base.py", line 70, in check_call
raise DGLError(py_str(_LIB.DGLGetLastError()))
dgl._ffi.base.DGLError: [13:01:20] /home/ubuntu/dgl_old/dgl/src/rpc/rpc.cc:489: Check failed: p_id < machine_count (8 vs. 8) : Invalid partition ID.
Stack trace:
[bt] (0) /usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/libdgl.so(+0x5ec8b8) [0x7f1c369d78b8]
[bt] (1) /usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/libdgl.so(+0x5efe0b) [0x7f1c369dae0b]
[bt] (2) /usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/libdgl.so(+0x5f1158) [0x7f1c369dc158]
[bt] (3) /usr/local/lib/python3.8/dist-packages/dgl-1.1-py3.8-linux-x86_64.egg/dgl/libdgl.so(DGLFuncCall+0x60) [0x7f1c36832760]
[bt] (4) /lib/x86_64-linux-gnu/libffi.so.7(+0x6ff5) [0x7f1c52c47ff5]
[bt] (5) /lib/x86_64-linux-gnu/libffi.so.7(+0x640a) [0x7f1c52c4740a]
[bt] (6) /usr/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(_ctypes_callproc+0x5b6) [0x7f1c52c60306]
[bt] (7) /usr/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x139dc) [0x7f1c52c609dc]
[bt] (8) /usr/bin/python3(_PyObject_MakeTpCall+0x296) [0x5f7056]
```

## Expected behavior

No crash

## Environment

- DGL Version (e.g., 1.0): 1.1
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): PyTorch 1.13.1+cpu
- OS (e.g., Linux): Ubuntu 20.04
- How you installed DGL (`conda`, `pip`, source): pip
- Build command you used (if compiling from source):
```
git clone https://github.com/dmlc/dgl.git
cd dgl
git submodule update --init --recursive
mkdir build
cd build
cmake ..
make -j32
```
- Python version: Python 3.8.10
- CUDA/cuDNN version (if applicable): n/a
- GPU models and configuration (e.g. V100): CPU Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
- Any other relevant information:
8 x r5.16xlarge AWS instances

## Additional context

The issue doesn't reproduce with tcmalloc

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.