huggingface / huggingface/optimum-executorch

bos_token_id and eos_token_id mismatch for optimum-internal-testing/tiny-random-llama

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
141
Forks
48
Avg merge
26m
Merged PRs (30d)
2

Description

When loading the cached model for inference with the example code below:
```python
tokenizer = AutoTokenizer.from_pretrained(model_id)
print(f"tokenizer: bos_token_id={tokenizer.bos_token_id}, eos_token_id={tokenizer.eos_token_id}")
model = ExecuTorchModelForCausalLM.from_pretrained(model_id, recipe="xnnpack", revision="executorch")
print(f"model: bos_token_id={model.bos_token_id}, eos_token_id={model.eos_token_id}")
```

I got:
```
File "/Users/guangyang/optimum-executorch/optimum/executorch/modeling.py", line 351, in text_generation
raise ValueError(
ValueError: The tokenizer's bos_token_id=1 must be the same as the model's bos_token_id=0.
```

Through the debug log, it shows "tokenizer: bos_token_id=1, eos_token_id=2" and "model: bos_token_id=0, eos_token_id=1". @echarlaix it seems like a bug in the tokenizer for this model.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.