alibaba / alibaba/graphlearn-for-pytorch
Errors in worker processes can cause entire program to hang.
- Dominant language
- Python
- Stars
- 146
- Forks
- 48
- PR merge metrics
- No merged PRs in 30d
Description
### 🐛 Describe the bug
See the below stack trace:
```
Process SpawnProcess-3:
Traceback (most recent call last):
File "/opt/conda/envs/glt/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/glt/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/glt/lib/python3.9/site-packages/graphlearn_torch/distributed/dist_sampling_producer.py", line 140, in _sampling_worker_loop
dist_sampler.sample_from_nodes(sampler_input[index])
File "/opt/conda/envs/glt/lib/python3.9/site-packages/graphlearn_torch/sampler/base.py", line 63, in __getitem__
return NodeSamplerInput(self.node[index], self.input_type)
IndexError: index 3 is out of bounds for dimension 0 with size 3
```
We should probably monitor the worker process health somewhere in _sampling_worker_loop` [1] and set `keep_running` false if they die.
[1]: https://github.com/alibaba/graphlearn-for-pytorch/blob/main/graphlearn_torch/python/distributed/dist_sampling_producer.py#L121-L163
### Environment
* GLT version: 26fe3d4e050b081bc51a79dc9547f244f5d314da
* PyG version:
* PyTorch version:
* OS:
* Python version:
* CUDA/cuDNN version:
* Any other relevant information
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked _sampling_worker_loop in graphlearn_torch/python/distributed/dist_sampling_producer.py and reproduce the provided worker traceback. Trace how worker failures affect keep_running and the parent process. Done means an error in a worker no longer leaves the entire program hanging, with coverage for the reported failure path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100