InternLM / InternLM/lmdeploy

Compatibility issue between InternLM-XComposer2-4KHD ChatTemplateConfig and BaseChatTemplate's messages2prompt method

Open
#2,092 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

I'm trying to use the ShareGPT4Video model with` InternLM-XComposer2-4KHD`, but I'm encountering format mismatches when passing query and image data to the messages2prompt method of BaseChatTemplate.
Is it correct that InternLM-XComposer2-4KHD's ChatTemplateConfig uses BaseChatTemplate's messages2prompt method? If so, the current implementation doesn't seem to handle the query and image format required for video processing models like ShareGPT4Video.
Have there been any recent changes or updates to this part of the codebase that might affect this functionality? If not, what would be the recommended approach to resolve this incompatibility

An error occurs in the following part
https://github.com/ShareGPT4Omni/ShareGPT4Video/blob/88426fd4a8386f3009368d424d5972881cdde311/captioner/slide_captioner_lmdeploy.py#L147-L148

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.