aigc-apps / aigc-apps/VideoX-Fun
Missing keys when loading Wan2.2-Fun-5B-Control in WanTransformer3DModel
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 188
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 3
Description
用 WanTransformer3DModel.from_pretrained(ckpt_dir, torch_dtype='torch.bfloat16').to(device) 加载 Wan2.2-Fun-5B-Control ,报missing keys :
['blocks.0.cross_attn.k_img.weight', 'blocks.0.cross_attn.k_img.bias', 'blocks.0.cross_attn.v_img.weight', 'blocks.0.cross_attn.v_img.bias', 'blocks.0.cross_attn.norm_k_img.weight', 'blocks.1.cross_attn.k_img.weight', 'blocks.1.cross_attn.k_img.bias', 'blocks.1.cross_attn.v_img.weight', 'blocks.1.cross_attn.v_img.bias', 'blocks.1.cross_attn.norm_k_img.weight', ...... , 'blocks.29.cross_attn.k_img.weight', 'blocks.29.cross_attn.k_img.bias', 'blocks.29.cross_attn.v_img.weight', 'blocks.29.cross_attn.v_img.bias', 'blocks.29.cross_attn.norm_k_img.weight']
查看源码和Wan2.2-Fun-5B-Control的内容发现,t2v_cross_attn 和 cross_attn 都是没问题的,而 i2v_cross_attn 中定义的 k_img 和 v_img 在 Wan2.2-Fun-5B-Control 中是不存在的,且 Wan2.2-Fun-5B-Control 的 config.json 的 "model_type" 是 "ti2v", 在 WanTransformer3DModel 中根据代码 cross_attn_type = 't2v_cross_attn' if model_type == 't2v' else 'i2v_cross_attn' 会以 i2v 加载cross_attention,但层keys不匹配了。
请问,是我使用的源码不对吗? 还是模型下载的不对? 感觉 WanTransformer3DModel 和 Wan2.2-Fun-5B-Control 好像并不匹配 。
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with WanTransformer3DModel and the Wan2.2-Fun-5B-Control config.json, then reproduce the reported from_pretrained call with the model checkpoint. Compare the selected cross-attention type with the checkpoint keys; done means the compatibility problem is resolved or the supported model/configuration mismatch is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100