Why is this tokenization weird?
Open
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
On My MacPro Sequoria, spacy 3.7.5 with en_core_web_trf model, the following sentence:
`How should writes to T012K and T012T be handled?`
Is tokenized as:
```
T012K => T012 and K
T012T => T012 and T
```
Other tokens look fine. this is a very normal sentence and how can it split them?
Contributor guide
Assessment
This issue has not been assessed yet.