DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA2
Can't hear the audio
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
import sys
sys.path.append('./')
from videollama2 import model_init, mm_infer
from videollama2.utils import disable_torch_init
def inference():
disable_torch_init()
# Video Inference
modal = 'video'
modal_path = '/data/video-llama2-av/VideoLLaMA2-audio_visual/assets/00001.mp4'
instruct = 'What exactly did the person in the video say?'
model_path = '/data/video-llama2-av/av-weight/VideoLLaMA2.1-7B-AV'
model, processor, tokenizer = model_init(model_path)
output = mm_infer(processor[modal](modal_path), instruct, model=model, tokenizer=tokenizer, do_sample=False, modal=modal)
print(output)
if __name__ == "__main__":
inference()
The output is: The person in the video spoke a few words, but they were not audible.
I input a video with sound, but it seems the model didn't pick it up. Is it because the audio branch isn't functioning properly? Also, I changed "mm_audio_tower" in VideoLLaMA2.1-7B-AV/config.json to the provided BEATs_iter3_plus_AS2M_finetuned_on_AS2M_cpt2.pt. Is this the correct place to make the change? Thanks for your reply!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.