codertimo / codertimo/BERT-pytorch
Vocab Replace \t to blank issue
Open
bug
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
when the corpus is:
` how are you \ tnice to meet you`
and apply `bert-vocab ` cmd, the output of the vacab is
`['', '', '', '', '', 'you', 'are', 'how', 'meet', 'nice', 'to']`.
But when change the corputs to
` how are you\tnice to meet you`, the result is `['', '', '', '', '', 'are', 'how', 'meet', 'to', 'you', 'younice']`, the last token become `younice`.
a <'blank'> need on both sides of <'\t'>.
it's may not a bug.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.