huggingface / huggingface/sharp-transformers
Feature request: Unigram tokenizer
Open
- Dominant language
- C#
- Stars
- 200
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
Unigram tokenizers are used in the best multilingual models e.g. multilingual-e5, paraphrase-multilingual-MiniLM-L12-v2.
It would be very helpful if you could add an implementation to expand the functionality to different languages.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.