Lightning-AI / Lightning-AI/pytorch-lightning
"MisconfigurationException: No supported gpu backend found!" with multi gpu training in jupyter notebooks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
When trying to train on two GPUs in a jupyter notebooks environment on jarvislabs.ai with `ddp_notebooks` I get the following error "MisconfigurationException: No supported gpu backend found!".
I'm trying to train on two RTX 5000 GPUs. On a Kaggle GPU the same code runs without any problem.
Any ideas?
### How to reproduce the bug
```python
trainer = pl.Trainer(
max_epochs=2,
accelerator="gpu",
devices=2,
precision=16,
accumulate_grad_batches=2
)
trainer.fit(model, train_dl, val_dl)
```
### Error messages and logs
"MisconfigurationException: No supported gpu backend found!"
### Environment
```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0): 1.7.7
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 1.10): 1.11
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version: V11.6.55
#- GPU models and configuration: 2x RTX 5000
#- How you installed Lightning(`conda`, `pip`, source): pip
#- Running environment of LightningApp (e.g. local, cloud): jarvislabs.ai
```
### More info
_No response_
cc @justusschock @awaelchli
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the shown Trainer configuration with ddp_notebooks on jarvislabs.ai using two RTX 5000 GPUs, then compare it with the working Kaggle environment and the reported Python, CUDA, and Lightning versions. Done means multi-GPU notebook training starts without the “No supported gpu backend found!” exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter, python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100