aigc-apps / aigc-apps/VideoX-Fun
Why not mask padding token when calculate latent - text cross attention?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 188
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 3
Description
Hi,
In the code here:
https://github.com/aigc-apps/VideoX-Fun/blob/a17b35acfbeb8f1ed30db3ad1be84d1b22ab05da/videox_fun/models/wan_transformer3d.py#L1010
I found when calculating the cross attention between latent and text condition, the context lens is set to None. Which means padding tokens will be included when calculate cross attention.
I double checked the attention score. I found the attention scores of padding tokens is much larger than other tokens:
For example, in the example below, the text_len is set to 40. tokens after 27 position are padding tokens. Please note that I manually multiply -1 for sort. So you can see padding tokens have larger scores.
So is it a bug?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Inspect videox_fun/models/wan_transformer3d.py around line 1010 and trace how the latent-to-text cross-attention receives context_lens. Compare the handling of padded tokens with the attention implementation and determine whether masking is expected; done requires a confirmed explanation or an agreed change, with validation in the relevant attention path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100