Compatibility issue between InternLM-XComposer2-4KHD ChatTemplateConfig and BaseChatTemplate's messages2prompt method
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
I'm trying to use the ShareGPT4Video model with` InternLM-XComposer2-4KHD`, but I'm encountering format mismatches when passing query and image data to the messages2prompt method of BaseChatTemplate.
Is it correct that InternLM-XComposer2-4KHD's ChatTemplateConfig uses BaseChatTemplate's messages2prompt method? If so, the current implementation doesn't seem to handle the query and image format required for video processing models like ShareGPT4Video.
Have there been any recent changes or updates to this part of the codebase that might affect this functionality? If not, what would be the recommended approach to resolve this incompatibility
An error occurs in the following part
https://github.com/ShareGPT4Omni/ShareGPT4Video/blob/88426fd4a8386f3009368d424d5972881cdde311/captioner/slide_captioner_lmdeploy.py#L147-L148
Contributor guide
Assessment
This issue has not been assessed yet.