zhanghang1989 / zhanghang1989/PyTorch-Encoding
Exception: process 0 terminated with signal SIGKILL`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2k
- Forks
- 448
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
dependency:
pytorch 1.4.0,
CUDA 10.2
Pytorch_encoding master branch
the following code is run on single GPU( GeForce RTX 2080, 8GB):
CUDA_VISIBLE_DEVICES=0 python train_dist.py --dataset PContext --model EncNet --aux --se-loss
the following issue happened:
Using poly LR scheduler with warm-up epochs of 0!
Starting Epoch: 0
Total Epoches: 80
Traceback (most recent call last):
File "train_dist.py", line 319, in <module>
main()
File "train_dist.py", line 148, in main
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
File "/home/qyuan/anaconda3/envs/pytorch_encoding_interpreter/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/home/qyuan/anaconda3/envs/pytorch_encoding_interpreter/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 107, in join
(error_index, name)
Exception: process 0 terminated with signal SIGKILL
I have read all the issues related to it, but the issue is still there,could you please give me some suggestions? thanks for your time!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the command in the issue with PyTorch 1.4.0, CUDA 10.2, and a single GeForce RTX 2080. Start in train_dist.py around main() lines 148 and 319, follow the mp.spawn path into main_worker, and use the traceback and process termination to identify the cause; done means training completes without process 0 receiving SIGKILL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100