dotnet / dotnet/machinelearning

text tokenizer EncodeToTokens repeat

Open
#7,523 2 comments 0 reactions 0 assignees View on GitHub
area-Tokenizers untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

input text："如果此时在位检测传感器的橙色LED灯熄灭"
Encodes input text to a list of EncodedTokens. offset value repeat

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.