Megvii-BaseDetection / Megvii-BaseDetection/YOLOX
When device=2, Exception: process 0 terminated with signal SIGKILL
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Description
I can run with d=1, but when d=2
Traceback (most recent call last):
File "tools/train.py", line 125, in
launch(
File "/home/user_name/Project/YOLOX/yolox/core/launch.py", line 82, in launch
mp.start_processes(
File "/home/username/anaconda3/envs/yolox-0826/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/home/username/anaconda3/envs/yolox-0826/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 105, in join
raise Exception(
Exception: process 0 terminated with signal SIGKILL
Exception ignored in: <function COCODataset.del at 0x7effec4f4b80>
Traceback (most recent call last):
File "/home/username/Project/YOLOX/yolox/data/datasets/coco.py", line 62, in del
del self.imgs
AttributeError: imgs
Traceback (most recent call last):
File "", line 1, in
File "/home/username/anaconda3/envs/yolox-0826/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/home/username/anaconda3/envs/yolox-0826/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated
(yolox-0826) username@username-Desktop:~/Project/YOLOX$ python tools/train.py -f exps/default/yolox_tiny.py -c yolox_tiny.pth -d 2 -b 1/home/username/anaconda3/envs/yolox-0826/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 39 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
^C

Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported command with -d 1 and -d 2, then start at tools/train.py and yolox/core/launch.py where multiprocessing is launched. Inspect yolox/data/datasets/coco.py alongside the SIGKILL, truncated-pickle, and leaked-semaphore errors. Done means the multi-device training command completes without these failures, or the resource constraint and required configuration are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100