OpenGVLab / OpenGVLab/Ask-Anything

Questions about VideoChat2_HD

Open
#194 35 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.4k
Forks
268
PR merge metrics
No merged PRs in 30d

Description

Hi, thanks for your update of VideoChat2_HD! When trying the newly-released code, I got some questions:

  • The MetaLoader_rs class in "train_it_ds.py" seems to be missing.
  • So I still used "train_it.py", but got the following error. I'm not sure whether it could be solved by using MetaLoader_rs.
RuntimeError: stack expects each tensor to be equal size, but got [8, 3, 224, 448] at entry 0 and [8, 3, 448, 672] at entry 1
  • Then I changed the batch_size to 1 and solved the previous error. But it seems the load_and_transform_media_data_image function does not have dynamic_config setting, which is passed to it in "it_dataset_mistral.py". I created a pull request to modify this part.
  • Is there any place to find the newly added dataset for VideoChat2_HD? I suppose the datasets are important to improve model performances.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing train_it_ds.py and train_it.py, then inspect it_dataset_mistral.py where dynamic_config is passed to load_and_transform_media_data_image. Reproduce the reported batching error and check where the VideoChat2_HD dataset is documented or loaded; done means the missing class, batch-size failure, and dynamic_config question have clear resolutions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.