huggingface / huggingface/swift-transformers
Improve tokenizer testing
Open
tokenization
- Dominant language
- Swift
- Stars
- 1.4k
- Forks
- 209
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 4
Description
- Retrieve tokenized results from a dataset instead of adding to the repo.
- Maybe create script to regenerate the reference baseline.
- Add more cases.
- Verify edge cases.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.