[BUG] vLLM and Transformers scores are different for the same model and same task

I am evaluating the same model using `community_tasks/arabic_evals.py` with both vLLM and transformers as backend but the scores are very different for the Alrage task which use LLM as judge.

- vLLM: `export VLLM_WORKER_MULTIPROC_METHOD=spawn && lighteval vllm model_name=tiiuae/Falcon-H1-3B-Base,dtype=auto,trust_remote_code=False,tensor_parallel_size=1,gpu_memory_utilization=0.8
--custom-tasks 'community|alrage_qa|0|0' --use-chat-template` ALRAGE=0.1784
- Transformers: `accelerate launch {accelerate_args} lighteval accelerate --multi_gpu --num_processes=1 -m model_name=tiiuae/Falcon-H1-3B-Base,dtype=auto,batch_size=1,trust_remote_code=False,model_parallel=False --custom-tasks 'community|alrage_qa|0|0' --use-chat-template` ALRAGE=0.2568
I am using the default values for both and tensor_parallel_size: 1 for vLLM