Strange Performance issues with PL + FFCV

Nobody has claimed this yet.

## 🐛 Bug

Hey there. I'm having some very weird performance issues when trying to use PL in combination with FFCV for ImageNet.

Here are some of the weird results I'm getting:

| Pure for loops (200 batches) | time |
| ---------------------------- | -------------- |
| PyTorch | 0:00:59.851324 |
| FFCV | 0:00:17.056751 |

| Training on 50 batches | time |
| -------------------------- | -------------- |
| Manual loop (FFCV) | 0:00:47.665201 |
| Manual loop (Pytorch) | 0:00:32.682359 |
| PL Trainer + DataLoaders | 0:00:33.111235 |
| PL Trainer + FFCV | **0:03:05.346748** |
| PL Trainer + Obfuscated DataLoader | 0:00:31.623927 |

As you can see, there is a very weird slowdown that only happens when using FFCV in conjunction with PL. However, iterating over the ffcv Loader is faster than the PyTorch loaders (as is expected), and the manual training loop, while a bit slower, is still in a comparable range.

Here is the source code:
- ImagenetFfcvDataModule: https://gist.github.com/lebrice/37d89c29388d7fc9ce267eed1ba6dbda
- The benchmarking script: https://gist.github.com/lebrice/8b00ea2911d88fcc6b3f9c156f49b555

The ImagenetDataModule is basically the same as in pl_bolts, with some customized prepare_data that copies the dataset from where it is on our cluster at Mila, to the fast SLURM_TMPDIR directory. The code for it can be found here: https://gist.github.com/lebrice/4a67df47d9fca3e199d3e7686396240c

I've got several questions. Does someone with FFCV experience in the PL team (e.g. @carmocca ) have any idea what might be going on here?
- I've considered the possibility that PL just assumes that the dataloaders give back CPU tensors, or that PL is somehow performing an extra copy from GPU to CPU. This doesn't appear to be the case though, as far as I can tell.
- I then thought that perhaps PL is doing some optimizations on the DataLoader it receives, which would explain the difference in runtime. However, even if the dataloader is obfuscated, (wrapped in functions, etc), the runtime stays the same (Btw, this is really great, really shows that you can pass arbitrary iterables to PL, not just DataLoaders).
- Perhaps the Trainer is having some effect on the FFCV side, e.g. by setting some global state, or initializing a torch.distributed context, that somehow affects the FFCV Loader implicitly?

### To Reproduce

1. Install FFCV
2. Pip install the ImagenetFfcvDataModule: `pip install git+https://gist.github.com/lebrice/37d89c29388d7fc9ce267eed1ba6dbda`
3. Run the benchmarking script above

### Expected behavior

I'd expect PyTorch-Lightning's Trainer to work well with FFCV.

### Environment

Details
* CUDA:
- GPU:
- Quadro RTX 8000
- available: True
- version: 10.2
* Lightning:
- lightning-bolts: 0.5.0
- pytorch-lightning: 1.7.0
- pytorch-pfn-extras: 0.5.8
- torch: 1.12.1
- torchmetrics: 0.7.3
- torchvision: 0.13.1
* Packages:
- absl-py: 1.2.0
- aiohttp: 3.8.1
- aiosignal: 1.2.0
- assertpy: 1.1
- async-timeout: 4.0.2
- attrs: 22.1.0
- autoflake: 1.4
- black: 22.6.0
- braceexpand: 0.1.7
- cachetools: 5.2.0
- certifi: 2022.6.15
- cfgv: 3.3.1
- charset-normalizer: 2.1.0
- click: 8.1.3
- cupy: 11.0.0
- cycler: 0.11.0
- distlib: 0.3.5
- fairscale: 0.4.6
- fastargs: 1.2.0
- fastrlock: 0.8
- ffcv: 0.0.3
- filelock: 3.7.1
- fonttools: 4.34.4
- frozenlist: 1.3.0
- fsspec: 2022.7.1
- google-auth: 2.9.1
- google-auth-oauthlib: 0.4.6
- grpcio: 1.47.0
- identify: 2.5.2
- idna: 3.3
- imgcat: 0.5.0
- importlib-metadata: 4.12.0
- iniconfig: 1.1.1
- joblib: 1.1.0
- kiwisolver: 1.4.4
- lightning-bolts: 0.5.0
- llvmlite: 0.38.1
- markdown: 3.4.1
- markupsafe: 2.1.1
- matplotlib: 3.5.2
- multidict: 6.0.2
- mypy: 0.910
- mypy-extensions: 0.4.3
- nodeenv: 1.7.0
- numba: 0.55.2
- numpy: 1.22.4
- oauthlib: 3.2.0
- opencv-python: 4.6.0
- packaging: 21.3
- pandas: 1.4.3
- pathspec: 0.9.0
- pillow: 9.2.0
- pip: 22.1.2
- platformdirs: 2.5.2
- pluggy: 1.0.0
- pre-commit: 2.20.0
- protobuf: 3.19.4
- psutil: 5.9.0
- py: 1.11.0
- pyasn1: 0.4.8
- pyasn1-modules: 0.2.8
- pydeprecate: 0.3.2
- pyflakes: 2.5.0
- pyparsing: 3.0.9
- pytest: 7.1.2
- python-dateutil: 2.8.2
- pytorch-lightning: 1.7.0
- pytorch-pfn-extras: 0.5.8
- pytz: 2022.1
- pyyaml: 6.0
- requests: 2.28.1
- requests-oauthlib: 1.3.1
- rsa: 4.9
- scikit-learn: 1.1.1
- scipy: 1.9.0
- setuptools: 61.2.0
- simple-parsing: 0.0.20
- six: 1.16.0
- sklearn: 0.0
- tensorboard: 2.9.1
- tensorboard-data-server: 0.6.1
- tensorboard-plugin-wit: 1.8.1
- terminaltables: 3.1.10
- threadpoolctl: 3.1.0
- toml: 0.10.2
- tomli: 2.0.1
- torch: 1.12.1
- torchmetrics: 0.7.3
- torchvision: 0.13.1
- tqdm: 4.64.0
- typing-extensions: 4.1.1
- urllib3: 1.26.11
- virtualenv: 20.16.2
- webdataset: 0.2.5
- werkzeug: 2.2.1
- wheel: 0.37.1
- yarl: 1.8.0
- zipp: 3.8.1
* System:
- OS: Linux
- architecture:
- 64bit
- ELF
- processor: x86_64
- python: 3.9.13
- version: #189-Ubuntu SMP Wed May 18 14:13:57 UTC 2022

Any help would be greatly appreciated!

cc @justusschock @awaelchli @ninginthecloud @rohitgr7 @otaj @borda @akihironitta

Start by installing FFCV and running the linked benchmarking script with the ImagenetFfcvDataModule gist, using the reported Lightning, Torch, and CUDA versions where possible. Compare the manual, Trainer, DataLoader, and obfuscated-loader timings, then identify and document the cause of the FFCV-specific slowdown; the issue is resolved when Trainer performance is comparable to the other paths.

Written by the indexing model from the issue text.