bert_tokenizer_params in Subword tokenizers guide
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.3k
- Forks
- 379
- Avg merge
- 3h 30m
- Merged PRs (30d)
- 8
Description
I follow the guide to Generate the vocabulary step and modify bert_tokenizer_params to
bert_tokenizer_params=dict(max_bytes_per_word=42, max_chars_per_token=9)
base on text.BertTokenizer docs cause the bert_vocab_args code has a comment mention that this is the arguments for text.BertTokenizer.
bert_vocab_args = dict(
...
# Arguments for `text.BertTokenizer`
bert_tokenizer_params=bert_tokenizer_params,
...
)
Then I use bert_vocab_args to generate the vocab file. But It shows error, turn out the bert_tokenizer_params in bert_vocab_args is the input of bert_tokenizer.BasicTokenizer() which doesn't have max_bytes_per_word nor max_chars_per_token in the __init__() of BasicTokenizer code.
I update the bert_tokenizer_params inside the bert_vocab_args to an empty dict
bert_vocab_args = dict(
...
# Arguments for `text.BertTokenizer`
bert_tokenizer_params={},
...
)
and move the bert_tokenizer_params=dict(max_bytes_per_word=42, max_chars_per_token=9) to Build the tokenizer step. I'm still waiting bert_vocab_from_dataset() code run and see how my vocab file look like and how my tokenizer behave.
I think you should clearly state which are the text.BertTokenizer params and which are the BasicTokenizer params or update the bert_vocab_from_dataset.py code to match what the guide means.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Subword tokenizers guide sections on generating the vocabulary and building the tokenizer, then compare their parameter examples with bert_vocab_from_dataset.py and bert_tokenizer.py. Confirm which arguments belong to BasicTokenizer versus text.BertTokenizer, and document the distinction or align the guide and code so the example runs without the reported error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100