huggingface / huggingface/optimum-intel

Phi-3.5-vision-instruct invalid EOS token

Open
#1,630 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
620
Forks
270
Avg merge
3d 10h
Merged PRs (30d)
22

Description

Hello,
I have a problem with exporting the model microsoft/Phi-3.5-vision-instruct.
Model loads and generates correctly, but it doesn't stop after answering and generates random output unrelated to the prompt.
After some troubleshooting I found that EOS token id in generation_config.json is 2, while it should be 32000. After changing it manually it works correctly.
Export command:
```
optimum-cli export openvino --model microsoft/Phi-3.5-vision-instruct --weight-format int4 --sym --ratio 1.0 --group-size -1 --trust-remote-code models\microsoft\Phi-3.5-vision-instruct
```
I've used optimum from main with transformers==4.53.3 (https://github.com/huggingface/optimum-intel/issues/1573).

Python: 3.10.7

Kind regards

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.