DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA3

How to prepare fine-tuning data for the temporal grounding task?

Open
#54 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
89
PR merge metrics
No merged PRs in 30d

Description

I noticed that you use the YouCook2 dataset for temporal grounding. When I prepare the data in this format, I modify the stage4_2b.sh script and replace the data_path with the YouCook2 jsonl file for fine-tuning. However, I encountered an issue where the process fails to start due to the lack of data_dict["conversations"]. It seems that the YouCook2 dataset doesn't have it. Could you please clarify how the data format for temporal grounding is handled on your side, especially how the temporal information is incorporated into the labels?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with stage4_2b.sh and the YouCook2 JSONL input, then trace where data_dict["conversations"] is required. Done means documenting the accepted fine-tuning format and explaining how temporal information is represented in the labels, with a working preparation path if the repository provides one.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
data, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.