BENCHmarking new flash attention!

Congrats on Flash Attention in the latest version, or to be precise, in having your storage limit increased on Pypi.org so you could upload the release that was weeks ago. Here are some benchmarks for your edification!

Testing Setup

• RTX 4090, Windows 10
• Used the generate_batch method with various beam sizes. The model was unloaded and reloaded each time - i.e. no caching of static prompt. Tokens per second, however, is only calculated on compute time not model loading time.
• Quantization int8 used. I DID NOT use float16/bfloat16 and then quant at runtime. Rather, I converted to int8 before runtime in order to remove that variable.
• Tested 10 times using flash attention and not. The numbers are the averages of those 10 runs.

Observations:

• Small but noticeable increase in tokens per second at beam size = 1. Disappointing at first...
• HOWEVER, as beam size increases, using flash attention DRAMATICALLY mitigates the typical decrease in tokens per second associated with more beams.
• MOREOVER, using flash attention DRAMATICALLY mitigates the increase in VRAM usage associated with more beams.

Implications

• This identical SOLAR model in GGUF Q8_0 sits at 40.92 tokens per second using 14,541.75 MB of VRAM. It's been clear that GGUF always uses more VRAM the the Q8/int8 comparison. Moreover, GGUF remains faster...SORT OF...
• GGUF does not have a beam size parameter. Transformers + bitsandbytes does (but I'll leave that for another day).
• At beam size 1 and using flash attention, the tokens per second increased by roughly 18% and the required VRAM decreased by approximately 4%. However, the benefits of flash attention are more pronounced as you increase beam size.
• THEREFORE, you can get a higher degree of accuracy with a smaller model by increasing beam size.

For example:

In my RAG application I love the Zephyr 3b model. If I put beams at 5, it religiously produces the same quality of results as my other favorite for RAG: neural-chat 7B (using a single beam). However, using 5 beams with the Zephyr 3b model puts its VRAM usage to MORE than the neural-chat model (again, which uses 1 beam). Therefore, there is no point in using Zephyr 3b since I can simply use neural-chat 7b.

Now, however, I can can use Zephyr 3B with 5 beams to achieve the same quality of neural-chat 7b (with 1 beam), but with less VRAM.

In conclusion, GGUF is "sort of" faster because it's only faster if you disregard Ctranslate2's ability to boost the quality of smaller models to one tier up by using beam size + flash attention, which, in the process, dramatically reduces VRAM usage and will inevitably increase tokens per second as well.

Thus, it's not appropriate to do a direct comparison of int8 and GGUF Q8_0 as of late. Here's a graph for you visual people. Feel free to contact me with any questions! Also, this is only a hobby so I'm always open to critiques.

No files, tests, or entry points are named; the benchmark setup and attached graph are the available evidence. There is no requested change or definition of done, so a contributor cannot begin implementation from this issue alone.