facebookresearch / facebookresearch/fairseq2

How to register original tokenizer

Open
#1,238 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

I tried to train a Transformer base model using a SentencePiece model tokenizer I downloaded from IWSLT, but I encountered the following error:

ERROR fairseq2 - The 'iwslt_spm' text tokenizer has an unknown family 'basic_sentencepiece'

The documentation didn't seem to have instructions on how to register and use my own tokenizer. Could you please show me how to do it?

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by locating the tokenizer registration and configuration documentation in fairseq2, then verify the documented steps against the reported 'basic_sentencepiece' error. Done means a newcomer can register and use an IWSLT SentencePiece tokenizer without that error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.