facebookresearch / facebookresearch/fairseq2
How to register original tokenizer
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
I tried to train a Transformer base model using a SentencePiece model tokenizer I downloaded from IWSLT, but I encountered the following error:
ERROR fairseq2 - The 'iwslt_spm' text tokenizer has an unknown family 'basic_sentencepiece'
The documentation didn't seem to have instructions on how to register and use my own tokenizer. Could you please show me how to do it?
Contributor guide
Research direction
No file or test is named. Start by locating the tokenizer registration and configuration documentation in fairseq2, then verify the documented steps against the reported 'basic_sentencepiece' error. Done means a newcomer can register and use an IWSLT SentencePiece tokenizer without that error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100