iOS Llama demo app segfaults if the tokenizer file is malformed
Open
@larryliu0820 is already working on this.
Since May 14, 2025.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
🐛 Describe the bug
For example, this happens if you use stories110M with the tokenizer.bin in the llama2.c repo instead of converting it from tokenizer.model with our tools. We should handle malformed input gracefully.
Versions
main
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.