huggingface / huggingface/optimum-intel

Difference in WWB Scores for LTX2 Text2Video compared to HF when using transformers v4/v5

Open
#1,998 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
620
Forks
270
Avg merge
3d 10h
Merged PRs (30d)
22

Description

I checked 3 way WWB scores from PR openvinotoolkit/openvino.genai#4438 for LTX2 Text-to-video. The results were drastically different depending on the transformers version used.

1 sample, CPU, fp32.

| Comparison | transformers 5 export | transformers 4 export |
|---|---|---|
| Optimum vs HF | 0.7615 | 0.9770 |
| GenAI vs HF | 0.7436 | 0.9841 |
| GenAI vs Optimum | 0.9734 | 0.9858 |

GenAI (transformers 4 export) vs the transformers 5 HF reference: 0.9841 (identical to the transformers 4 HF reference, so the torch output is unchanged across versions; the divergence is in the transformers 5 export).

Setup:
diffusers==0.39.0 for both
transformers==4.57.6 for v4, and transformers==5.0.0 for v5.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the WWB comparisons using the transformers 4.57.6 and 5.0.0 setups, then inspect openvinotoolkit/openvino.genai#4438 and the LTX2 Text-to-video export path. Compare the resulting exports against the HF references; done means the transformers 5 export no longer causes the reported score divergence.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.