Megvii-BaseDetection / Megvii-BaseDetection/YOLOX

When device=2, Exception: process 0 terminated with signal SIGKILL

Open
#593 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
10.6k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Description

I can run with d=1, but when d=2

Traceback (most recent call last):
File "tools/train.py", line 125, in
launch(
File "/home/user_name/Project/YOLOX/yolox/core/launch.py", line 82, in launch
mp.start_processes(
File "/home/username/anaconda3/envs/yolox-0826/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/home/username/anaconda3/envs/yolox-0826/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 105, in join
raise Exception(
Exception: process 0 terminated with signal SIGKILL
Exception ignored in: <function COCODataset.del at 0x7effec4f4b80>
Traceback (most recent call last):
File "/home/username/Project/YOLOX/yolox/data/datasets/coco.py", line 62, in del
del self.imgs
AttributeError: imgs
Traceback (most recent call last):
File "", line 1, in
File "/home/username/anaconda3/envs/yolox-0826/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/home/username/anaconda3/envs/yolox-0826/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated
(yolox-0826) username@username-Desktop:~/Project/YOLOX$ python tools/train.py -f exps/default/yolox_tiny.py -c yolox_tiny.pth -d 2 -b 1/home/username/anaconda3/envs/yolox-0826/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 39 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
^C

image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported command with -d 1 and -d 2, then start at tools/train.py and yolox/core/launch.py where multiprocessing is launched. Inspect yolox/data/datasets/coco.py alongside the SIGKILL, truncated-pickle, and leaked-semaphore errors. Done means the multi-device training command completes without these failures, or the resource constraint and required configuration are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.