InternLM / InternLM/lmdeploy

[Feature] 请问是否可以用qwen2.5-vl的video frame的相邻帧优化

Open
#3,714 1 comment 0 reactions 1 assignee Claimed by @CUHKSZzxy View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

*### Motivation

如题,参考https://lmdeploy.readthedocs.io/zh-cn/latest/multi_modal/qwen2_5_vl.html
进行视频帧推理时,是否能正常使用qwen2.5-vl的video frame相邻帧优化?还是把所有帧都作为单图token进行使用。

我的代码如下:
`
question = ''
*# 视频封面图
cover_img = None
question = question + f'cover image:\n{IMAGE_TOKEN}\n\n'

*# 视频帧
imgs = []
question = question + 'visual frames:\n'
for i in range(len(imgs)):
question = question + f'Frame{i+1}: {IMAGE_TOKEN}\n'

*# ques
question += 'What are the cover and visuals of this video talking about?'

*# content
content = [{'type': 'text', 'text': question}]
content.append({'type': 'image_url', 'image_url': {"url": f"{cover_img}"}})
for img in imgs:
content.append({'type': 'image_url', 'image_url': {'url': f"{img}"}})

messages = [dict(role='user', content=content)]
`

如果上述无法实现我的目标作用的话,是否有其它wiki参考进行实现?

期待您的回复,感谢!

### Related resources

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.