huggingface / huggingface/optimum-intel
Difference in WWB Scores for LTX2 Text2Video compared to HF when using transformers v4/v5
- Dominant language
- Jupyter Notebook
- Stars
- 620
- Forks
- 270
- Avg merge
- 3d 10h
- Merged PRs (30d)
- 22
Description
I checked 3 way WWB scores from PR openvinotoolkit/openvino.genai#4438 for LTX2 Text-to-video. The results were drastically different depending on the transformers version used.
1 sample, CPU, fp32.
| Comparison | transformers 5 export | transformers 4 export |
|---|---|---|
| Optimum vs HF | 0.7615 | 0.9770 |
| GenAI vs HF | 0.7436 | 0.9841 |
| GenAI vs Optimum | 0.9734 | 0.9858 |
GenAI (transformers 4 export) vs the transformers 5 HF reference: 0.9841 (identical to the transformers 4 HF reference, so the torch output is unchanged across versions; the divergence is in the transformers 5 export).
Setup:
diffusers==0.39.0 for both
transformers==4.57.6 for v4, and transformers==5.0.0 for v5.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the WWB comparisons using the transformers 4.57.6 and 5.0.0 setups, then inspect openvinotoolkit/openvino.genai#4438 and the LTX2 Text-to-video export path. Compare the resulting exports against the HF references; done means the transformers 5 export no longer causes the reported score divergence.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100