zhanghang1989 / zhanghang1989/PyTorch-Encoding

Exception: process 0 terminated with signal SIGKILL`

Open
#324 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2k
Forks
448
PR merge metrics
No merged PRs in 30d

Description

Hi,

dependency:
pytorch 1.4.0,
CUDA 10.2
Pytorch_encoding master branch

the following code is run on single GPU( GeForce RTX 2080, 8GB):
CUDA_VISIBLE_DEVICES=0 python train_dist.py --dataset PContext --model EncNet --aux --se-loss

the following issue happened:

Using poly LR scheduler with warm-up epochs of 0! Starting Epoch: 0 Total Epoches: 80 Traceback (most recent call last): File "train_dist.py", line 319, in <module> main() File "train_dist.py", line 148, in main mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args)) File "/home/qyuan/anaconda3/envs/pytorch_encoding_interpreter/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn while not spawn_context.join(): File "/home/qyuan/anaconda3/envs/pytorch_encoding_interpreter/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 107, in join (error_index, name) Exception: process 0 terminated with signal SIGKILL

I have read all the issues related to it, but the issue is still there,could you please give me some suggestions? thanks for your time!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the command in the issue with PyTorch 1.4.0, CUDA 10.2, and a single GeForce RTX 2080. Start in train_dist.py around main() lines 148 and 319, follow the mp.spawn path into main_worker, and use the traceback and process termination to identify the cause; done means training completes without process 0 receiving SIGKILL.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.