DAMO-NLP-SG/Video-LLaMA

View on GitHub

[EMNLP 2023 Demo] Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

Stars: 3.1k
Forks: 286
Open beginner issues: 0
Indexed issues: 63

Dominant language: Python
License: BSD-3-Clause
Last GitHub push: Jun 4, 2024
Latest indexed: Sep 15, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Open All

63 open issues indexed

Great job! Will you release the machine-translated VideoChat instructions data? Open

DAMO-NLP-SG/Video-LLaMA#5 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Could you please share the code to generate the instruct data? Open

DAMO-NLP-SG/Video-LLaMA#49 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
finetune-billa7b-zh模型推理错误 Open

DAMO-NLP-SG/Video-LLaMA#53 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
FileNotFoundError: Could not find module 'C:\Users\USER\miniconda3\envs\videollama\Lib\site-packages\torchaudio\lib\libtorchaudio.pyd' (or one of its dependencies). Try using the full path with constructor syntax. Open

DAMO-NLP-SG/Video-LLaMA#61 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
demo运行 Open

DAMO-NLP-SG/Video-LLaMA#67 · 9 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
考虑加一下deepspeed吗？ Open

DAMO-NLP-SG/Video-LLaMA#71 · 1 comment · 2 reactions · 0 assignees · Indexed Sep 15, 2026
可以看一下最终的文件树结构吗？ Open

DAMO-NLP-SG/Video-LLaMA#72 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
可以提供一个infer的脚本吗？非gradio demo的那种 Open

DAMO-NLP-SG/Video-LLaMA#77 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Training model Open

DAMO-NLP-SG/Video-LLaMA#81 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
visionbranch stage2收敛问题 Open

DAMO-NLP-SG/Video-LLaMA#85 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
visionbranch_stage1预训练webvid数据集问题 Open

DAMO-NLP-SG/Video-LLaMA#86 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Failed to run Billa7B-SFT and error complains wrong shape to reshape position_ids Open

DAMO-NLP-SG/Video-LLaMA#88 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Error(s) in loading state_dict for VideoLLAMA Open

DAMO-NLP-SG/Video-LLaMA#97 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Output Logit Open

DAMO-NLP-SG/Video-LLaMA#98 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
After completing the fine-tuning of 'visionbranch stage2', what should be its loss value to be considered as having correctly converged? Open

DAMO-NLP-SG/Video-LLaMA#100 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
运行demo,需要多大的内存 Open

DAMO-NLP-SG/Video-LLaMA#101 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
预训练的参数设置问题 Open

DAMO-NLP-SG/Video-LLaMA#102 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Time taken Open

DAMO-NLP-SG/Video-LLaMA#104 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
inferece如何使用多张V100代替一张A100？ Open

DAMO-NLP-SG/Video-LLaMA#107 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Repeating itself Open

DAMO-NLP-SG/Video-LLaMA#109 · 2 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
Difference between the Self-implemented BLIP2 vs HF version? Open

DAMO-NLP-SG/Video-LLaMA#114 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
nvrtc: error: invalid value for --gpu-architecture (-arch) Open

DAMO-NLP-SG/Video-LLaMA#116 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
7B模型，video微调训练时，Dataloader获取数据获取不到 Open

DAMO-NLP-SG/Video-LLaMA#117 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
请问训练设置val，这样正确吗 Open

DAMO-NLP-SG/Video-LLaMA#118 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
关于environment.yml文件的问题 Open

DAMO-NLP-SG/Video-LLaMA#120 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
如何部署LLama2训练出的video llama？ Open

DAMO-NLP-SG/Video-LLaMA#121 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Do you have any plans to open-source the pre-training and fine-tuning checkpoints based on Llama 2 Chinese version? Open

DAMO-NLP-SG/Video-LLaMA#125 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
change the frames and query_tokens size Open

DAMO-NLP-SG/Video-LLaMA#128 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Multiple Video-Text pair Support Open

DAMO-NLP-SG/Video-LLaMA#129 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Prompt Open

DAMO-NLP-SG/Video-LLaMA#132 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
How to finetune video-llama using deepspeed? Open

DAMO-NLP-SG/Video-LLaMA#133 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM: size mismatch for model.embed_tokens.weight: copying a param with shape torch.Size([32001, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]). size mismatch for lm_head.weight: copying a param with shape torch.Size([32001, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]). Open

DAMO-NLP-SG/Video-LLaMA#135 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Dear author, How much time does it cost to train this model？ With what type of GPU cards? Open

DAMO-NLP-SG/Video-LLaMA#136 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
inf value occurs during forwarding process when fine-tuning VL branch with LLAVA-150K+MiniGPT4-3.5K+webvid-instruct Open

DAMO-NLP-SG/Video-LLaMA#138 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
example model deployment Open

DAMO-NLP-SG/Video-LLaMA#139 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
A demo without gradio Open

DAMO-NLP-SG/Video-LLaMA#140 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
multi-cards training Open

DAMO-NLP-SG/Video-LLaMA#141 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Incorrect model inference (what went wrong in my setup) Open

DAMO-NLP-SG/Video-LLaMA#145 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
What is the input sample of the forward function in videollama Open

DAMO-NLP-SG/Video-LLaMA#146 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
如何提升下游任务上finetune的效果 Open

DAMO-NLP-SG/Video-LLaMA#147 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Unable to launch demo Open

DAMO-NLP-SG/Video-LLaMA#149 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Is video-LLaMA capable of comprehending videos that have faces surrounded by bounding boxes(face recognition) Open

DAMO-NLP-SG/Video-LLaMA#150 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Evaluation on large-scale dataset Open

DAMO-NLP-SG/Video-LLaMA#151 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Compatibility b/w torch and torchvision? Open

DAMO-NLP-SG/Video-LLaMA#152 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Possible bugs in LR scheduler Open

DAMO-NLP-SG/Video-LLaMA#154 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
how to increase the numbers of input frame? Open

DAMO-NLP-SG/Video-LLaMA#155 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
What if no frame_position_embeddings? Open

DAMO-NLP-SG/Video-LLaMA#158 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
llm在两个阶段都是keep frozen吗？ Open

DAMO-NLP-SG/Video-LLaMA#160 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
finetune-billa7b-zh inference error shape '[-1, 136]' is invalid for input of size 137 Open

DAMO-NLP-SG/Video-LLaMA#161 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Finetune with LoRA and QLoRA Open

DAMO-NLP-SG/Video-LLaMA#162 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026

DAMO-NLP-SG/Video-LLaMA

Get new issues in your inbox