Seeking help in understanding dataset preprocessing

Hello,
I try to understand how you preprocessed the CodRep datasets for sequencer and I don't understand several things :

- I've cloned the CodRep competition repository and I don't have the same amount of data as you do. More precisely, I've concatenated dataset 1, 2, 3 & 5, and after removing duplicates, I still have 35767 instances. If I remove 5% for validation, I still have 33978 instances. You have 33798, Which other instances did you delete ? Same for dataset 4 which you used as test, After removing duplicates, I have 7097 instances but you have 4711.

- I understood the transformation of a source code to its abstract buggy context by using `abstraction-1.0-SNAPSHOT-jar-with-dependencies.jar` but I don't understand how you preprocessed data from abstract buggy context to the format of `src-train.txt` as I can see special token like ``. Same for `tgt-train.txt`. Are the scripts used for these tasks public ?

Thank you !

Start with the CodRep competition repository and inspect abstraction-1.0-SNAPSHOT-jar-with-dependencies.jar, src-train.txt, and tgt-train.txt. Compare the reported dataset counts and trace how abstract buggy contexts become the training files, including the token. Done means the filtering steps and preprocessing scripts are identified or documented.

Written by the indexing model from the issue text.