dmlc / dmlc/dgl

[GraphBolt] GraphBolt objects cannot be serialized during multiprocessing

Open
#5,953 0 comments 0 reactions 1 assignee Claimed by @RamonZhou View on GitHub
bug:confirmed
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

It seems that GraphBolt objects like `SampledSubgraph` cannot be passed around in multiprocessing due to lack of serialization method.

## To Reproduce

```python
import dgl.graphbolt
import torch
import multiprocessing as mp
import scipy.sparse as sp

def rand_csc_graph(N, density):
adj = sp.random(N, N, density)
adj = adj + adj.T
adj = adj.tocsc()

indptr = torch.LongTensor(adj.indptr)
indices = torch.LongTensor(adj.indices)

graph = dgl.graphbolt.from_csc(indptr, indices)

return graph

def _entry(graph, q):
adjs = []
seeds = torch.arange(5)

for hop in range(2):
sg = graph.sample_neighbors(seeds, torch.LongTensor([2]))
seeds = sg.indices
adjs.insert(0, sg)

print('Putting')
q.put(adjs)
print('Done put')

if __name__ == '__main__':
graph = rand_csc_graph(200, 0.15)
q = mp.Queue()
proc = mp.Process(target=_entry, args=(graph, q))
proc.start()
print('Getting')
item = q.get()
print('Done get')
proc.join()
print(item)
```

This code snippet hangs indefinitely, either silently or throws the following error:
```
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.9/multiprocessing/queues.py", line 245, in _feed
obj = _ForkingPickler.dumps(obj)
File "/home/ubuntu/miniconda3/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
RuntimeError: Tried to serialize object __torch__.torch.classes.graphbolt.SampledSubgraph which does not have a __getstate__ method defined!
```

## Expected behavior

The data should be successfully printed. Replacing `q.put(adjs)` to something else like `q.put(42)` works. So it should be something explicit to GraphBolt.

## Environment

- DGL Version (e.g., 1.0): master
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3):
- OS (e.g., Linux):
- How you installed DGL (`conda`, `pip`, source):
- Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version (if applicable):
- GPU models and configuration (e.g. V100):
- Any other relevant information:

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.