alibaba / alibaba/AliceMind

mplug中两处代码错误问题

Open
#91 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2k
Forks
301
PR merge metrics
No merged PRs in 30d

Description

AliceMind/mPLUG
/caption_mplug.py中234-239行中,无论是visual encoder用了ViT-B-16还是ViT-L-14,在根据输入图像分辨率更改positional embedding时,创建的pos_embedd的dim=1都是768。
```
# reshape positional embedding to accomodate for image resolution change
if config["clip_name"] == "ViT-B-16":
num_patches = int(config["image_res"] * config["image_res"]/(16*16))
elif config["clip_name"] == "ViT-L-14":
num_patches = int(config["image_res"] * config["image_res"]/(14*14))
pos_embed = nn.Parameter(torch.zeros(num_patches + 1, 768).float())

pos_embed = resize_pos_embed(state_dict['visual_encoder.visual.positional_embedding'].unsqueeze(0),
pos_embed.unsqueeze(0))
state_dict['visual_encoder.visual.positional_embedding'] = pos_embed
```
而在初始化MPLUG模型中初始化CLIP的时,即调用AliceMind/models
/visual_transformers.py中初始化CLIP代码时,positional embedding的dim=1的值时根据visual encoder用了ViT-B-16还是ViT-L-14而决定的,B-16时dim=1为768,L-14时dim=1为1024。
```
def initialize_clip(config, num_patches=240):
from models.clip import clip
if config["clip_name"] == "ViT-B-16":
clip_model, preprocess = clip.load("ViT-B-16.tar", jit=False)
num_patches = int(config['image_res']*config['image_res']/(16*16))
pos_embed = nn.Parameter(torch.zeros(num_patches + 1, 768).float())
elif config["clip_name"] == "ViT-L-14":
clip_model, preprocess = clip.load("ViT-L-14.tar", jit=False)
num_patches = int(config['image_res']*config['image_res']/(14*14))
pos_embed = nn.Parameter(torch.zeros(num_patches + 1, 1024).float())
pos_embed.weight = resize_pos_embed(clip_model.visual.positional_embedding.unsqueeze(0), pos_embed.unsqueeze(0))
clip_model.visual.positional_embedding = pos_embed
return clip_model, preprocess
```
两代码的区别造成
`RuntimeError: Error(s) in loading state_dict for MPLUG: size mismatch for visual_encoder.visual.positional_embedding: copying a param with shape torch.Size([x, 768]) from checkpoint, the shape in current model is torch.Size([x, 1024]).`的错误。

如有错误敬请指正

Contributor guide

No contributing guide indexed for this repository

Research direction

Compare the positional-embedding initialization in caption_mplug.py lines 234-239 with initialize_clip in models/visual_transformers.py for both ViT-B-16 and ViT-L-14. Reproduce the reported state_dict loading error and verify that the embedding dimensions used for resizing and model loading are consistent for each visual encoder.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.