How to convert tokenizer of SmolLM model as accepted by executorch
Open
@guangy10 is already working on this.
Since Nov 13, 2024.
module: extension
module: user experience
triaged
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
Hi,
I am trying to convert SmolLm-135M-Instruct model to .pte format and then run on an android device.
I have been successful in converting the model but executorch requires the tokenizer in either .bin format or .model format which can then be converted into .bin format. But on huggingface tokenizer.model or tokenizer.bin files are not present.
How would I go about converting the tokenizer.json file into the appropriate format.
cc @mergennachin @byjlw
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.