dotnet / dotnet/machinelearning
text tokenizer EncodeToTokens repeat
Open
area-Tokenizers
untriaged
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
input text："如果此时在位检测传感器的橙色LED灯熄灭"
Encodes input text to a list of EncodedTokens. offset value repeat
Contributor guide
Research direction
Start by reproducing the reported behavior at the EncodeToTokens entry point with the supplied Chinese text and inspect the returned token offsets. Trace where offsets are assigned and add or run a regression check showing that repeated offsets are corrected for this input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100