huggingface / huggingface/transformers.js-examples
Tokenizer playground: whitespace handling
Open
- Dominant language
- JavaScript
- Stars
- 2.1k
- Forks
- 251
- PR merge metrics
- No merged PRs in 30d
Description
Models such as BERT and T5 do not render their whitespace.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the tokenizer playground entry point and reproduce the issue with BERT and T5, using the attached image as the expected context. Done means whitespace is visibly represented for models that currently do not render it; no test or file path is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- frontend, web-dev
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100