dotnet / dotnet/machinelearning

text tokenizer EncodeToTokens repeat

Open
#7,523 2 comments 0 reactions 0 assignees View on GitHub
area-Tokenizers untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

input text:"如果此时在位检测传感器的橙色LED灯熄灭"
Encodes input text to a list of EncodedTokens. offset value repeat

Image

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported behavior at the EncodeToTokens entry point with the supplied Chinese text and inspect the returned token offsets. Trace where offsets are assigned and add or run a regression check showing that repeated offsets are corrected for this input.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.