JetBrains-Research / JetBrains-Research/code2seq

'\n' mixed in Vocabulary['token']

Open
#111 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
62
Forks
17
PR merge metrics
No merged PRs in 30d

Description

it seems that counter in vocabulary is counting 'token' tokens with a newline character.
for example, vocabulary.pkl in java-small dataset, i can find
'return': 6020684,
and
'return\n': 33290,
separately.

i personally fixed this problem by stripping path_context on Vocabulary._process_raw_sample,
but im little confused whether this problem(mixing '\n' in tokens) is intended.

thank you!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.