image diffusion model output changes for the worse after 530ff92655
Open
bot-triaged
module: cublas
module: cuda
module: numerical-stability
triaged
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.6k
- PR merge metrics
- PR metrics pending
Description
I'm not sure if this is acceptable numerical drift or not. I've been bisecting a change where an image diffusion workflow in SGLang produces the following images from PyTorch 2.10 -> PyTorch 2.11. This is one of the issues blocking SGLang from upgrading to PyTorch 2.11.
I bisected this change down to 530ff92655. Claude tells me that this PR makes some more cases go through cublaslt, which we do know we have some problems with recently (@eqy) Any thoughts here? @ngimel @drisspg @nikitaved
GOOD: (approx PyTorch 2.10)
BAD: (approx PyTorch 2.11)
cc @ptrblck @msaroufim @eqy @jerryzh168 @tinglvv @nWEIdia @csarofeen
Contributor guide
Assessment
This issue has not been assessed yet.