JetBrains-Research / JetBrains-Research/code2seq
'\n' mixed in Vocabulary['token']
Open
- Dominant language
- Python
- Stars
- 62
- Forks
- 17
- PR merge metrics
- No merged PRs in 30d
Description
it seems that counter in vocabulary is counting 'token' tokens with a newline character.
for example, vocabulary.pkl in java-small dataset, i can find
'return': 6020684,
and
'return\n': 33290,
separately.
i personally fixed this problem by stripping path_context on Vocabulary._process_raw_sample,
but im little confused whether this problem(mixing '\n' in tokens) is intended.
thank you!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.