Slower inference results for BLOOM fp16 on identical hardware

Hey,

Thank you for the scripts for loading checkpoints and running benchmarks. I have a strange issue that ds_inference fp16 throughput is quite slower than the results mentioned. But, the int8 benchmark results are almost identical.

Environment:
GCP `a2-ultragpu-8g` with A100 8x80GB, 1.3 TB Memory, 96 vCPUs
Debian 11

For fp16 & batch size 1, the throughput I receive is 67 msecs/token while it should be possible to get 44 msecs/token. This trend is repeated for higher batch sizes too.

But for int8, the results are exactly the same as the one mentioned in benchmarks (both for 8x80GB and 4x80GB).

What have I tried until now?
1. Different CUDA versions (11.0, 11.4, 11.6, 11.7), PyTorch versions, DeepSpeed versions (0.7.0, 0.7.2, 0.7.3)
2. Reinstalling environment from scratch on a new server

Any idea on what I might be doing wrong? Or is everybody else experiencing similar throughput?

Iniziare dagli script utilizzati per caricare i checkpoint ed eseguire i benchmark ds_inference, confrontando il percorso fp16 con il percorso int8 con batch size pari a 1 e con batch size maggiori. Riprodurre i 67 msecs/token riportati sulla configurazione A100 indicata, quindi determinare se l’implementazione o l’ambiente spiega la differenza rispetto al risultato documentato di 44 msecs/token.

Scritto dal modello di indicizzazione a partire dal testo della issue.