[Benchmarks] index

This issue is to document the important `transformers` benchmarks in one place, so that they are easy to find.

To add a new benchmark entry post it in an Issue (separately or as a comment in an existing issue) and then link from here. If you have edit rights please add a link directly to this post, otherwise please add a note in the comments and I will update this post.

Please do not post actual benchmarks in the comments of this Issue. This is only an index.

Thank you!

## Fastest speed combinations
- [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1005229426)
- [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005227577)

## Precision: fp16 vs bf16 vs tf32 vs fp32
- [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803)
- [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189)

## Batch size / gradient accumulation steps
- gradient accumulation steps: [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537), [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004592231)
- batch size [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004470417), [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005033957)

## Gradient checkpointing
- [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004422281)
- [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005034578)

## Optimizers:
- Adam torch vs. apex vs HF vs adafactor: [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1005219385), [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005220263)
- re-run the above a year later with the same list of optimizers, plus BNB's 8bit optimizer and fused torch AdamW [PCIe 80GB A100](https://github.com/huggingface/transformers/issues/22101)

## Network / Interconnects:
- [DP/DDP/NVLink](https://github.com/huggingface/transformers/issues/9371#issuecomment-768656711)