dotnet / dotnet/machinelearning
Consider Rust BPE tokenizer implementation
Open
area-Tokenizers
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
Try to look at rust gems BPE tokenizer performance and figure out if it is worth adapting it in the ML tokenizers library.
https://github.com/github/rust-gems/blob/main/crates/bpe/README.md
Contributor guide
Assessment
This issue has not been assessed yet.