codertimo / codertimo/BERT-pytorch

Vocab Replace \t to blank issue

Open
#33 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
6.5k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

when the corpus is:
` how are you \ tnice to meet you`
and apply `bert-vocab ` cmd, the output of the vacab is
`['', '', '', '', '', 'you', 'are', 'how', 'meet', 'nice', 'to']`.
But when change the corputs to
` how are you\tnice to meet you`, the result is `['', '', '', '', '', 'are', 'how', 'meet', 'to', 'you', 'younice']`, the last token become `younice`.
a <'blank'> need on both sides of <'\t'>.
it's may not a bug.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.