facebookresearch / facebookresearch/detectron2

introduce torch.compile in DDP mode cause abnormal terminated with signal SIGSEGV

Open
#5,325 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

## Instructions To Reproduce the Issue:
to speedup training, I add torch.compile operation after DistributedDataParallel in detectron2/engine/defaults.py:

```
ddp = DistributedDataParallel(model, **kwargs)
ddp = torch.compile(ddp, mode="max-autotune")
```
And I trained ViTDet model, it terminated abnormally with following exception message:

```
Traceback (most recent call last):
File "/home/jupyterhub/proshm/detectron2/tools/lazyconfig_train_net.py", line 124, in
launch(
File "/home/jupyterhub/proshm/detectron2/detectron2/engine/launch.py", line 69, in launch
mp.start_processes(
File "/home/ps/miniconda3/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 202, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/home/ps/miniconda3/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 145, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 3 terminated with signal SIGSEGV
```
## Environment:
I installed detectron2 in ubuntu with pytorch 2.1

Contributor guide

Open the contributing guide

Research direction

Start with the DDP setup in detectron2/engine/defaults.py and the launch path through tools/lazyconfig_train_net.py and detectron2/engine/launch.py. Reproduce the ViTDet training setup on Ubuntu with PyTorch 2.1 using the reported torch.compile placement, then isolate the SIGSEGV. Done means distributed training no longer terminates abnormally in this configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.