dotnet / dotnet/machinelearning
text tokenizer EncodeToTokens repeat
Open
area-Tokenizers
untriaged
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
input text:"如果此时在位检测传感器的橙色LED灯熄灭"
Encodes input text to a list of EncodedTokens. offset value repeat
Contributor guide
Assessment
This issue has not been assessed yet.