Lightning-AI / Lightning-AI/pytorch-lightning
ddp_fork encounters ProcessExitedException: process 1 terminated with signal SIGSEGV
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
Hi,
I have been experiencing `ProcessExitedException: process 1 terminated with signal SIGSEGV` error when using `ddp_fork` with multi-gpus. I have done some research on the potential reasons and read through this PR (https://github.com/Lightning-AI/lightning/pull/18132), but I think mine is a different problem as I have put the data loading code inside the ``*_dataloader()`` hooks in the`LightningModule`.
This is a blocker for users who want to do multi-gpu training in notebook.
### How to reproduce the bug
```
import pytorch_lightning as pl
import torch
from torch import nn
import torch.nn.functional as F
from timm import create_model
import torchvision
import torchvision.transforms as transforms
class ToyLitModel(pl.LightningModule):
def __init__(self, model):
super().__init__()
self.model = model
self.criterion = nn.CrossEntropyLoss()
def forward(self, x):
return self.model(x)
def training_step(self, batch, batch_idx):
x, y = batch
x = F.interpolate(x, size=224)
logits = self(x)
loss = self.criterion(logits, y)
return loss
def validation_step(self, batch, batch_idx):
x, y = batch
logits = self(x)
loss = self.criterion(logits, y)
return loss
def configure_optimizers(self):
return torch.optim.Adam(self.model.parameters(), lr=0.001)
def prepare_data(self) -> None:
train_set = torchvision.datasets.CIFAR10(
root="~/data", train=True, download=True
)
def train_dataloader(self):
transform = transforms.Compose(
[transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]
)
batch_size = 32
train_set = torchvision.datasets.CIFAR10(
root="~/data", train=True, transform=transform
)
# train_set = torch.utils.data.Subset(train_set, list(range(1000)))
train_loader = torch.utils.data.DataLoader(
train_set, batch_size=batch_size, shuffle=True, num_workers=4
)
return train_loader
def run():
model = create_model("swin_base_patch4_window7_224", num_classes=10)
# model = create_model("resnet18", num_classes=10)
task = ToyLitModel(model)
trainer = pl.Trainer(max_epochs=1, strategy="ddp_fork", devices=2)
trainer.fit(task)
if __name__ == "__main__":
run()
```
### Error messages and logs
```
ProcessExitedException Traceback (most recent call last)
Cell In[4], line 6
4 # train_loader, val_loader = load_data()
5 trainer = pl.Trainer(max_epochs=1, strategy="ddp_notebook", devices=2, enable_progress_bar=True)
----> 6 trainer.fit(task)
File ~/anaconda3/envs/test-install/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:532, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
530 self.strategy._lightning_module = model
531 _verify_strategy_supports_compile(model, self.strategy)
--> 532 call._call_and_handle_interrupt(
533 self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
534 )
File ~/anaconda3/envs/test-install/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py:42, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
40 try:
41 if trainer.strategy.launcher is not None:
---> 42 return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
43 return trainer_fn(*args, **kwargs)
45 except _TunerExitException:
File ~/anaconda3/envs/test-install/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py:127, in _MultiProcessingLauncher.launch(self, function, trainer, *args, **kwargs)
119 process_context = mp.start_processes(
120 self._wrapping_function,
121 args=process_args,
(...)
124 join=False, # we will join ourselves to get the process references
125 )
126 self.procs = process_context.processes
--> 127 while not process_context.join():
128 pass
130 worker_output = return_queue.get()
File ~/anaconda3/envs/test-install/lib/python3.9/site-packages/torch/multiprocessing/spawn.py:140, in ProcessContext.join(self, timeout)
138 if exitcode < 0:
139 name = signal.Signals(-exitcode).name
--> 140 raise ProcessExitedException(
141 "process %d terminated with signal %s" %
142 (error_index, name),
143 error_index=error_index,
144 error_pid=failed_process.pid,
145 exit_code=exitcode,
146 signal_name=name
147 )
148 else:
149 raise ProcessExitedException(
150 "process %d terminated with exit code %d" %
151 (error_index, exitcode),
(...)
154 exit_code=exitcode
155 )
ProcessExitedException: process 1 terminated with signal SIGSEGV
```
### Environment
- python=3.9.18
- pytorch-lightning==2.0.8
- torch==2.0.1
- torchvision==0.15.2
- timm==0.9.7
### More info
_No response_
cc @justusschock @awaelchli
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the notebook scenario with the supplied CIFAR10 example, using the `ddp_fork` strategy and two GPUs. Start at `strategies/launchers/multiprocessing.py`, particularly `_MultiProcessingLauncher.launch`, and trace the worker termination; done means the same multi-GPU training run completes without `ProcessExitedException` or SIGSEGV.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100