ByteDance-Seed / ByteDance-Seed/VideoWorld
Question about reproducing CALVIN results with released calvin_model.pth
- Dominant language
- Python
- Stars
- 801
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
Hi authors,
Thank you for releasing VideoWorld and the CALVIN checkpoint.
I tried to reproduce the CALVIN results using the released `calvin_model.pth` and the README evaluation pipeline. However, my results are quite different from the paper for some task categories.
Using the released setup, I obtained roughly:
- Open/Close: 73.7%
- Push: 2.3%
- Turn on/off: 31.6%
In Table 2 of the paper, the reported results are:
- Open/Close: 75.4%
- Push: 56.2%
- Turn on/off: 72.1%
So Open/Close is close, but Push and Turn on/off are much lower.
Could you please clarify:
1. Is the released `calvin_model.pth` expected to reproduce the CALVIN results in the paper?
2. Is there any missing checkpoint / training stage / evaluation detail that is not included in the public release?
3. How is the IDM (or action head) for CALVIN trained in the paper setting? Is that part already included in `calvin_model.pth`?
I would really appreciate any clarification. Thank you.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.