huggingface / huggingface/tokenizers
.NET bindings
- Dominant language
- Rust
- Stars
- 11k
- Forks
- 1.2k
- Avg merge
- 3d 8h
- Merged PRs (30d)
- 26
Description
Greetings 😊 and thank you all for open-sourcing the state-of-the-art tokenizer library!
A few months ago, I've implemented .NET(C#, F#, ...) bindings of 🤗 Tokenizers and wanted to share it here:
https://github.com/sappho192/Tokenizers.DotNet
Some basic features (encode, decode, loading tokenizer) are done and tested, and generally available via [NuGet package manager](https://github.com/sappho192/Tokenizers.DotNet?tab=readme-ov-file#nuget-package-list).
The API structure does not mimic the original (Rust) so far, but in the near future I plan to provide all the functionality of the original implementation.
If you have any advice or feedback, please let me know!
Contributor guide
Assessment
This issue has not been assessed yet.