using tk.sentencepeicetokenizer for my trained tokeinzer
Open
- Dominant language
- Jupyter Notebook
- Stars
- 113
- Forks
- 22
- PR merge metrics
- No merged PRs in 30d
Description
Hello, HOw can I use a toeknizer that I have trained which is in .json file to be used using your sentencepeicetoeknizer to be able to do the evulation that you have considered ? Please I need help in this
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue mentions a trained tokenizer in a .json file, SentencePiece tokenization, and evaluation, but names no files, tests, or entry points. First inspect the repository's tokenizer and evaluation usage, then clarify the expected integration and define completion as successfully evaluating with the trained tokenizer.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100