modelscope / modelscope/DiffSynth-Studio
H100 GPU - FP8 inference acceleration
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
Thanks for the great work !
I followed the Qwen-Image inference example in below link (recommended for faster inference) for comparing FP8 vs bfloat16:
./accelerate/Qwen-Image-FP8.py
But for FP8 (torch.float8_e4m3fn), the inference is slower than bfloat16.
FP8 takes 42s per image with a single-gpu instance and num_inference_steps=40
bfloat16 takes 33s under the same setting.
Setup:
- H100 GPU which supports FP8
- Flash Attention 3 is available
How to improve FP8's inference speed?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with examples/qwen_image/accelerate/Qwen-Image-FP8.py and run the reported FP8 and bfloat16 configurations on an H100. Compare the per-image timings and inspect the inference path and available Flash Attention 3 setup. Done means identifying and addressing the cause of FP8 being slower, with measurements showing improved FP8 performance under the same settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100