bytedance / bytedance/Dolphin

Bug Report: Pretrained checkpoint and current DonutModel mismatch

Open
#147 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9k
Forks
775
PR merge metrics
No merged PRs in 30d

Description

Hello,

**Describe the bug**
I cloned the latest master (commit 7df02e13, Sept 30, 2025) and downloaded the pre-trained model via Google Drive (`dolphin_model.bin` and `dolphin_tokenizer.json`).

When loading the pretrained weights, the model fails to load into the current `DonutModel` due to size mismatch errors in the vision encoder layers.

It seems that the current `DonutModel` vision tower (`vpm.*` layers) has different dimensions compared to the released checkpoint.

**Questions**
- Are the Google Drive weights outdated relative to the current code?
- Could you release an updated checkpoint matching the latest master?

Thanks a lot for your work on this project!

**Error traceback**

```
RuntimeError: Error(s) in loading state_dict for DonutModel:
size mismatch for vpm.model.layers.1.downsample.norm.weight: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([512]).
size mismatch for vpm.model.layers.1.downsample.norm.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([512]).
size mismatch for vpm.model.layers.1.downsample.reduction.weight: copying a param with shape torch.Size([512, 1024]) from checkpoint, the shape in current model is torch.Size([256, 512]).
size mismatch for vpm.model.layers.2.downsample.norm.weight: copying a param with shape torch.Size([2048]) from checkpoint, the shape in current model is torch.Size([1024]).
size mismatch for vpm.model.layers.2.downsample.norm.bias: copying a param with shape torch.Size([2048]) from checkpoint, the shape in current model is torch.Size([1024]).
size mismatch for vpm.model.layers.2.downsample.reduction.weight: copying a param with shape torch.Size([1024, 2048]) from checkpoint, the shape in current model is torch.Size([512, 1024]).
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the released dolphin_model.bin and dolphin_tokenizer.json with the current DonutModel vision tower, especially the vpm.model.layers.1 and layers.2 shapes shown in the traceback. Check whether the checkpoint matches another revision of the model; done means establishing a compatible checkpoint/code pairing or documenting that the released weights are outdated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.