Differences between BugFixTokenPairs files

In the data folder, there are two types of datasets, one is named BugFixNoDup_ and BugFixTokenPairs_commits. Can you explain the difference between these two types of datasets? If I want to re-run the experiment for pre-training on BugFix data then which dataset should I be using.

Start in the data folder by comparing the BugFixNoDup_ and BugFixTokenPairs_commits datasets, then trace the pre-training experiment entry point to see which one it loads. Done means documenting how the datasets differ and clearly identifying the dataset to use for rerunning BugFix pre-training.

Written by the indexing model from the issue text.