ByteDance-Seed / ByteDance-Seed/VideoWorld

Question about reproducing CALVIN results with released calvin_model.pth

Open
#17 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
801
Forks
40
PR merge metrics
No merged PRs in 30d

Description

Hi authors,

Thank you for releasing VideoWorld and the CALVIN checkpoint.

I tried to reproduce the CALVIN results using the released `calvin_model.pth` and the README evaluation pipeline. However, my results are quite different from the paper for some task categories.

Using the released setup, I obtained roughly:

- Open/Close: 73.7%
- Push: 2.3%
- Turn on/off: 31.6%

In Table 2 of the paper, the reported results are:

- Open/Close: 75.4%
- Push: 56.2%
- Turn on/off: 72.1%

So Open/Close is close, but Push and Turn on/off are much lower.

Could you please clarify:

1. Is the released `calvin_model.pth` expected to reproduce the CALVIN results in the paper?
2. Is there any missing checkpoint / training stage / evaluation detail that is not included in the public release?
3. How is the IDM (or action head) for CALVIN trained in the paper setting? Is that part already included in `calvin_model.pth`?

I would really appreciate any clarification. Thank you.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.