alibaba / alibaba/graphlearn-for-pytorch

Errors in worker processes can cause entire program to hang.

Open
#156 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
146
Forks
48
PR merge metrics
No merged PRs in 30d

Description

### 🐛 Describe the bug

See the below stack trace:

```
Process SpawnProcess-3:
Traceback (most recent call last):
File "/opt/conda/envs/glt/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/glt/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/envs/glt/lib/python3.9/site-packages/graphlearn_torch/distributed/dist_sampling_producer.py", line 140, in _sampling_worker_loop
dist_sampler.sample_from_nodes(sampler_input[index])
File "/opt/conda/envs/glt/lib/python3.9/site-packages/graphlearn_torch/sampler/base.py", line 63, in __getitem__
return NodeSamplerInput(self.node[index], self.input_type)
IndexError: index 3 is out of bounds for dimension 0 with size 3
```

We should probably monitor the worker process health somewhere in _sampling_worker_loop` [1] and set `keep_running` false if they die.

[1]: https://github.com/alibaba/graphlearn-for-pytorch/blob/main/graphlearn_torch/python/distributed/dist_sampling_producer.py#L121-L163

### Environment

* GLT version: 26fe3d4e050b081bc51a79dc9547f244f5d314da
* PyG version:
* PyTorch version:
* OS:
* Python version:
* CUDA/cuDNN version:
* Any other relevant information

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked _sampling_worker_loop in graphlearn_torch/python/distributed/dist_sampling_producer.py and reproduce the provided worker traceback. Trace how worker failures affect keep_running and the parent process. Done means an error in a worker no longer leaves the entire program hanging, with coverage for the reported failure path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.