Lightning-AI / Lightning-AI/pytorch-lightning

ddp_fork encounters ProcessExitedException: process 1 terminated with signal SIGSEGV

Open
#18,541 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

3rd party bug strategy: ddp ver: 2.0.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description
Hi,

I have been experiencing `ProcessExitedException: process 1 terminated with signal SIGSEGV` error when using `ddp_fork` with multi-gpus. I have done some research on the potential reasons and read through this PR (https://github.com/Lightning-AI/lightning/pull/18132), but I think mine is a different problem as I have put the data loading code inside the ``*_dataloader()`` hooks in the`LightningModule`.

This is a blocker for users who want to do multi-gpu training in notebook.

### How to reproduce the bug

```
import pytorch_lightning as pl
import torch
from torch import nn
import torch.nn.functional as F
from timm import create_model
import torchvision
import torchvision.transforms as transforms

class ToyLitModel(pl.LightningModule):
def __init__(self, model):
super().__init__()
self.model = model
self.criterion = nn.CrossEntropyLoss()

def forward(self, x):
return self.model(x)

def training_step(self, batch, batch_idx):
x, y = batch
x = F.interpolate(x, size=224)
logits = self(x)
loss = self.criterion(logits, y)
return loss

def validation_step(self, batch, batch_idx):
x, y = batch
logits = self(x)
loss = self.criterion(logits, y)
return loss

def configure_optimizers(self):
return torch.optim.Adam(self.model.parameters(), lr=0.001)

def prepare_data(self) -> None:
train_set = torchvision.datasets.CIFAR10(
root="~/data", train=True, download=True
)

def train_dataloader(self):
transform = transforms.Compose(
[transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]
)

batch_size = 32

train_set = torchvision.datasets.CIFAR10(
root="~/data", train=True, transform=transform
)
# train_set = torch.utils.data.Subset(train_set, list(range(1000)))
train_loader = torch.utils.data.DataLoader(
train_set, batch_size=batch_size, shuffle=True, num_workers=4
)
return train_loader

def run():
model = create_model("swin_base_patch4_window7_224", num_classes=10)
# model = create_model("resnet18", num_classes=10)
task = ToyLitModel(model)
trainer = pl.Trainer(max_epochs=1, strategy="ddp_fork", devices=2)
trainer.fit(task)

if __name__ == "__main__":
run()
```

### Error messages and logs

```
ProcessExitedException Traceback (most recent call last)
Cell In[4], line 6
4 # train_loader, val_loader = load_data()
5 trainer = pl.Trainer(max_epochs=1, strategy="ddp_notebook", devices=2, enable_progress_bar=True)
----> 6 trainer.fit(task)

File ~/anaconda3/envs/test-install/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:532, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
530 self.strategy._lightning_module = model
531 _verify_strategy_supports_compile(model, self.strategy)
--> 532 call._call_and_handle_interrupt(
533 self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
534 )

File ~/anaconda3/envs/test-install/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py:42, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
40 try:
41 if trainer.strategy.launcher is not None:
---> 42 return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
43 return trainer_fn(*args, **kwargs)
45 except _TunerExitException:

File ~/anaconda3/envs/test-install/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py:127, in _MultiProcessingLauncher.launch(self, function, trainer, *args, **kwargs)
119 process_context = mp.start_processes(
120 self._wrapping_function,
121 args=process_args,
(...)
124 join=False, # we will join ourselves to get the process references
125 )
126 self.procs = process_context.processes
--> 127 while not process_context.join():
128 pass
130 worker_output = return_queue.get()

File ~/anaconda3/envs/test-install/lib/python3.9/site-packages/torch/multiprocessing/spawn.py:140, in ProcessContext.join(self, timeout)
138 if exitcode < 0:
139 name = signal.Signals(-exitcode).name
--> 140 raise ProcessExitedException(
141 "process %d terminated with signal %s" %
142 (error_index, name),
143 error_index=error_index,
144 error_pid=failed_process.pid,
145 exit_code=exitcode,
146 signal_name=name
147 )
148 else:
149 raise ProcessExitedException(
150 "process %d terminated with exit code %d" %
151 (error_index, exitcode),
(...)
154 exit_code=exitcode
155 )

ProcessExitedException: process 1 terminated with signal SIGSEGV
```

### Environment

- python=3.9.18
- pytorch-lightning==2.0.8
- torch==2.0.1
- torchvision==0.15.2
- timm==0.9.7

### More info

_No response_

cc @justusschock @awaelchli

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the notebook scenario with the supplied CIFAR10 example, using the `ddp_fork` strategy and two GPUs. Start at `strategies/launchers/multiprocessing.py`, particularly `_MultiProcessingLauncher.launch`, and trace the worker termination; done means the same multi-GPU training run completes without `ProcessExitedException` or SIGSEGV.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.