Slow generation compared to transformers + PyTorch

I'm running the Llama example on a machine with an Nvidia T4 16GB to compare the performance with HF Transformers + PyTorch.

Here's the Python example I'm running:

```python
import time
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
model = AutoModelForCausalLM.from_pretrained(
'meta-llama/Llama-2-7b-hf',
torch_dtype=torch.float16,
).to('cuda')

prompt = '1' * 500
max_tokens = 200

start_time = time.time()
tokens = tokenizer(prompt, return_tensors='pt').to('cuda')
output = model.generate(
**tokens,
max_new_tokens=max_tokens,
temperature=2.0,
)
total_time = time.time() - start_time
generated_tokens = len(output[0, tokens['input_ids'].shape[1]:])
print(generated_tokens / total_time, 'tokens/s')
```

The command I'm using to test Candle's implementation is:

```bash
$ prompt=$(python3 -c 'print("1" * 500)')
$ cargo run --example llama --release --features "cuda" -- --prompt "${prompt}" --temperature 2.0 --sample-len 200 --dtype f16
loading the model weights from meta-llama/Llama-2-7b-hf
building the model
starting the inference loop
(...)
200 tokens generated (9.116397586726794 token/s)
```

- Python result: 14.3 tokens/s
- Candle result: 9.1 tokens/s

Using a batch size of 1 for both implementations, the speed difference is around ~57%, but I noticed this difference is greater when the batch size is >1.

Am I missing something? Perhaps the example includes gradient calculation by default or something similar?

Start with the Candle Llama example invoked by `cargo run --example llama --release --features "cuda"`, and reproduce the comparison using the supplied Python script and command. Trace the inference loop and compare its settings with Transformers + PyTorch, then establish whether the reported token-per-second gap and larger batch-size difference are expected or indicate a performance problem.

Written by the indexing model from the issue text.