能否支持glm-4v-9b模型
Open
planned feature
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Motivation
尝试适配glm-4v-9b模型(其实是13.9b,视觉部分有4.9B),发现glm4v里面对输入的position_ids做了特殊处理:
new_input_embeds.append(torch.cat(
(inputs_embeds[i, :boi_token_pos], images_features[i], inputs_embeds[i, eoi_token_pos + 1:])))
new_position_ids.append(torch.cat(
(position_ids[i, :boi_token_pos + 1], position_ids[i, boi_token_pos + 1].repeat(num_patches),
position_ids[i, eoi_token_pos:])
))
其将视觉特征部分的position_ids统一设定为同一个值,在计算RoPE时会用到。
turbomind引擎好像没有修改position_ids的接口。我们对glm-4v在我们的场景里全参数微调后效果是开源模型里面最好的,希望官方能支持glm-4v-9b模型
### Related resources
GLM-4模型链接:
https://github.com/THUDM/GLM-4
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.