dotnet / dotnet/machinelearning

[Tokenizers] How to load HF based tokenizers e.g. SmolLM

Open
#7,197 0 comments 1 reaction 1 assignee Claimed by @tarekgh View on GitHub
area-Tokenizers documentation enhancement question
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Often LLM models are distributed on HuggingFace or similar where tokenizers are presumed created via transformers library. This often contains a bunch of json/txt files. I have found it hard to then now how to create a ML.Tokenizer from that. For example how would one create a tokenizer for:

https://huggingface.co/HuggingFaceTB/SmolLM-135M-Instruct/tree/main

Could there be a getting started document detailing how to load tokenizers from such files and how to identify what to use to load these?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.