Slurm `gpus-per-task` Error

@awaelchli is already working on this.

Since Feb 21, 2023.

### Bug description

We recently uncovered what I believe is a PL bug in the way slurm multi-gpu is being handled. This seems to be caused by a change in slurm behavior for some recent versions (we haven't nailed down exactly when this change happened)

The error is caused by using the `--gpus-per-task` slurm argument (potentially other GPU arguments as well). For example if I have a machine with 8 GPUs I will say `--gpus-per-task=1` and `--tasks=8` to launch 8 processes with 1 GPU each. Previous versions of slurm didnt seem to enforce GPU isolation but the newer versions are setting `CUDA_VISIBLE_DEVICES` appropriately.

The problem is that lightning seems to expect to see all GPUs on a node from all processes, so it errors out with

```
MisconfigurationException('You requested gpu: [0, 1, 2, 3, 4, 5, 6, 7]\n But your machine only has: [0]
```

In this case, I think slurm is doing the right thing since this would prevent tasks from accidentally using GPUs that they werent assigned. I also think that PL shouldn't care if it can see multiple GPUs or not as long as all distributed processes check in with the primary. Since slurm is handling the multiprocessing, and PL knows this, it should only need to see a single GPU in order to work.

Is this a bug that should be fixed?

### How to reproduce the bug

Here's a redacted sbatch that reproduces

```
#!/bin/bash

# Parameters
#SBATCH --cpus-per-gpu=12
#SBATCH --gpus-per-task=1
#SBATCH --mem-per-gpu=112500
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --open-mode=append
#SBATCH --requeue
#SBATCH --signal=USR2@120
#SBATCH --time=240
```

The lightning command should be set up for 8 GPUs on 1 node.

### Error messages and logs

```
MisconfigurationException('You requested gpu: [0, 1, 2, 3, 4, 5, 6, 7]\n But your machine only has: [0]
```

### Environment

Current environment

```
* CUDA:
- GPU: None
- available: False
- version: 11.7
* Lightning:
- lightning-lite: 1.8.0
- lightning-utilities: 0.3.0
- pytorch-lightning: 1.8.0
- pytorch-msssim: 0.2.1
- torch: 1.13.0
- torch-fidelity: 0.3.0
- torchjpeg: 0.9.27
- torchmetrics: 0.10.2
- torchvision: 0.14.0
* Packages:
- absl-py: 1.3.0
- aiohttp: 3.8.3
- aiosignal: 1.2.0
- alembic: 1.8.1
- antlr4-python3-runtime: 4.9.3
- async-timeout: 4.0.2
- attrs: 22.1.0
- autopage: 0.5.1
- av: 8.1.0
- black: 22.10.0
- cachetools: 5.2.0
- certifi: 2022.9.24
- charset-normalizer: 2.1.1
- click: 8.1.3
- cliff: 4.0.0
- cloudpickle: 2.2.0
- cmaes: 0.8.2
- cmd2: 2.4.2
- colorlog: 6.7.0
- cython: 0.29.32
- docker-pycreds: 0.4.0
- fire: 0.4.0
- frozenlist: 1.3.1
- fsspec: 2022.10.0
- gitdb: 4.0.9
- gitpython: 3.1.29
- google-auth: 2.14.0
- google-auth-oauthlib: 0.4.6
- greenlet: 2.0.0
- grpcio: 1.50.0
- hydra-core: 1.2.0
- hydra-optuna-sweeper: 1.2.0
- hydra-submitit-launcher: 1.2.0
- idna: 3.4
- importlib-metadata: 5.0.0
- importlib-resources: 5.10.0
- lightning-lite: 1.8.0
- lightning-utilities: 0.3.0
- lmdb: 1.3.0
- lpips: 0.1.4
- mako: 1.2.3
- markdown: 3.4.1
- markupsafe: 2.1.1
- metabit: 0.0.0
- multidict: 6.0.2
- mypy-extensions: 0.4.3
- numpy: 1.23.4
- nvidia-cublas-cu11: 11.10.3.66
- nvidia-cuda-nvrtc-cu11: 11.7.99
- nvidia-cuda-runtime-cu11: 11.7.99
- nvidia-cudnn-cu11: 8.5.0.96
- oauthlib: 3.2.2
- omegaconf: 2.2.3
- opencv-python: 4.6.0.66
- opencv-python-headless: 4.6.0.66
- optuna: 2.10.1
- packaging: 21.3
- pathspec: 0.10.1
- pathtools: 0.1.2
- pbr: 5.11.0
- pillow: 9.3.0
- pip: 22.2.2
- platformdirs: 2.5.2
- prettytable: 3.5.0
- promise: 2.3
- protobuf: 3.19.6
- psutil: 5.9.3
- pyasn1: 0.4.8
- pyasn1-modules: 0.2.8
- pyflow: 1.0
- pyparsing: 3.0.9
- pyperclip: 1.8.2
- pytorch-lightning: 1.8.0
- pytorch-msssim: 0.2.1
- pyyaml: 6.0
- requests: 2.28.1
- requests-oauthlib: 1.3.1
- rsa: 4.9
- scipy: 1.9.3
- sentry-sdk: 1.10.1
- setproctitle: 1.3.2
- setuptools: 65.5.0
- shortuuid: 1.0.9
- six: 1.16.0
- smmap: 5.0.0
- sqlalchemy: 1.4.42
- stevedore: 4.1.0
- submitit: 1.4.5
- tensorboard: 2.10.1
- tensorboard-data-server: 0.6.1
- tensorboard-plugin-wit: 1.8.1
- termcolor: 2.1.0
- tomli: 2.0.1
- torch: 1.13.0
- torch-fidelity: 0.3.0
- torchjpeg: 0.9.27
- torchmetrics: 0.10.2
- torchvision: 0.14.0
- tqdm: 4.64.1
- typing-extensions: 4.4.0
- urllib3: 1.26.12
- wandb: 0.13.4
- wcwidth: 0.2.5
- werkzeug: 2.2.2
- wheel: 0.37.1
- yarl: 1.8.1
- zipp: 3.10.0
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.8.10
- version: #40~20.04.1-Ubuntu SMP Thu Jun 17 13:25:28 UTC 2021
```

cc @awaelchli