Why SentencePieceTokenizer can't save vocab file
- Dominant language
- Python
- Stars
- 746
- Forks
- 112
- PR merge metrics
- No merged PRs in 30d
Description
I want to use vocab file in PairedDataloader, but the the save_vocab function of SentencePieceTokenizer only save the model file.
The model file can't be load by Dataloader because of decoding error.
In [sentencepiece_tokenizer.py](https://github.com/asyml/texar-pytorch/blob/master/texar/torch/data/tokenizers/sentencepiece_tokenizer.py#L175), I saw you delete the vocab file.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in texar/torch/data/tokenizers/sentencepiece_tokenizer.py at the save_vocab function around line 175. Trace how PairedDataloader loads the vocabulary and reproduce the decoding error with the saved files. Done means save_vocab produces files that PairedDataloader can load without decoding errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100