facebookresearch / facebookresearch/detectron2
introduce torch.compile in DDP mode cause abnormal terminated with signal SIGSEGV
- Dominant language
- Python
- Stars
- 34.7k
- Forks
- 7.9k
- PR merge metrics
- No merged PRs in 30d
Description
## Instructions To Reproduce the Issue:
to speedup training, I add torch.compile operation after DistributedDataParallel in detectron2/engine/defaults.py:
```
ddp = DistributedDataParallel(model, **kwargs)
ddp = torch.compile(ddp, mode="max-autotune")
```
And I trained ViTDet model, it terminated abnormally with following exception message:
```
Traceback (most recent call last):
File "/home/jupyterhub/proshm/detectron2/tools/lazyconfig_train_net.py", line 124, in
launch(
File "/home/jupyterhub/proshm/detectron2/detectron2/engine/launch.py", line 69, in launch
mp.start_processes(
File "/home/ps/miniconda3/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 202, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/home/ps/miniconda3/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 145, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 3 terminated with signal SIGSEGV
```
## Environment:
I installed detectron2 in ubuntu with pytorch 2.1
Contributor guide
Research direction
Start with the DDP setup in detectron2/engine/defaults.py and the launch path through tools/lazyconfig_train_net.py and detectron2/engine/launch.py. Reproduce the ViTDet training setup on Ubuntu with PyTorch 2.1 using the reported torch.compile placement, then isolate the SIGSEGV. Done means distributed training no longer terminates abnormally in this configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100