Performance compared to lm-evaluation-harness

Hi,

Thanks for sharing this package, it has lots of cool features!

I saw that arc-challenge was taking about twice longer that what I have with harness, I ran the following commands with lighteval:

```bash
# ran with 4A100 GPUs -> 611s
time accelerate launch --multi_gpu --num_processes=4 lighteval/run_evals_accelerate.py --model_args="pretrained=meta-llama/Meta-Llama-3-8B" --tasks "leaderboard|arc:challenge|25|0" --output_dir "arc_challenge" --override batch_size 8
```
and the following command from harness (using big-refactor branch):
```bash
# ran with 4A100 GPUs -> 319s and big-refactor branch
time accelerate launch -m lm_eval \
--model hf \
--model_args pretrained=meta-llama/Meta-Llama-3-8B,dtype="bfloat16" \
--tasks arc_challenge \
--batch_size 8 \
--num_fewshot=25
```

Of-course, many things could cause this but I wanted to know if you have faced something similar or benchmarked light-eval compared to Harness?

If not, would you have a suggestion to get similar performance? (it seems bf16 are used by default so it should not be the culprit)