huggingface / huggingface/optimum-executorch
bos_token_id and eos_token_id mismatch for optimum-internal-testing/tiny-random-llama
- Dominant language
- Python
- Stars
- 141
- Forks
- 48
- Avg merge
- 26m
- Merged PRs (30d)
- 2
Description
When loading the cached model for inference with the example code below:
```python
tokenizer = AutoTokenizer.from_pretrained(model_id)
print(f"tokenizer: bos_token_id={tokenizer.bos_token_id}, eos_token_id={tokenizer.eos_token_id}")
model = ExecuTorchModelForCausalLM.from_pretrained(model_id, recipe="xnnpack", revision="executorch")
print(f"model: bos_token_id={model.bos_token_id}, eos_token_id={model.eos_token_id}")
```
I got:
```
File "/Users/guangyang/optimum-executorch/optimum/executorch/modeling.py", line 351, in text_generation
raise ValueError(
ValueError: The tokenizer's bos_token_id=1 must be the same as the model's bos_token_id=0.
```
Through the debug log, it shows "tokenizer: bos_token_id=1, eos_token_id=2" and "model: bos_token_id=0, eos_token_id=1". @echarlaix it seems like a bug in the tokenizer for this model.
Contributor guide
Assessment
This issue has not been assessed yet.