dmlc / dmlc/dgl

[Dist][Tensorpipe] program hangs sometimes during train

Open
#4,267 5 comments 1 reaction 1 assignee Assigned to @Rhett-Ying View on GitHub
bug:confirmed
Dominant language
Python
Stars
14.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

When train with `backend=tensorpipe`, program could hang sometimes, even after we fixed the hang issue caused by cython (previous issue ticket: https://github.com/dmlc/dgl/issues/3881). For now, no hang issue is hit when train with `backend=socket`.
## To Reproduce

Not confirmed yet and don't know how to reproduce it.

## Expected behavior

## Environment

- DGL Version (e.g., 1.0): master
- Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3):
- OS (e.g., Linux):
- How you installed DGL (`conda`, `pip`, source):
- Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version (if applicable):
- GPU models and configuration (e.g. V100):
- Any other relevant information:

## Additional context

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.