huggingface / huggingface/inference-benchmarker
Is it possible to use tekken.json instead of tokenizer.json?
Open
- Dominant language
- Rust
- Stars
- 172
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
Is it possible to use the `inference-bechmarker` with Mistral models without `tokenizer.json` (using `tekken.json`)?
For example, https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506.
Thanks.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.