The "Dump with files" option produces duplicate files

Todos

• Fix duplicate files generated when using dump with files
• Fix creation of multiple datasets/programs when multiple tasks use the same files
• Fix the task indexes in the yaml file when dump created

Details below by Johann-petrak

I have created a competition with input data, scoring programs from uploading a bundle file and there are also already test submissions.

When I then try to download using the "Dump with files" option I see the following oddities/problems:

• The zip file that I get contains several files with the same name, in my case this is input_data-6224.zip feference_data-6224.zip and scoring_program-6224.zip which all occur twice in the zip file with identical names.
• some of the reference*.zip files contain files which are definitely not from the project where I downloaded the dump from but from other projects I created (and deleted) previously
• I also seem to get different versions of the scoring program

What is going on here, should the dump not be restricted to those data/program files which are actually part of the project?

And if I upload a bundle that contains e.g. a target file or scorer and then remove the project, should those files and scorers not get removed with the project they belong to?

UPDATE: to avoid confusion, after further investigation, the problem here is only that the chose file names are duplicates, not the content, this was my mistake.

Start by reproducing the "Dump with files" workflow described in the issue with a competition containing uploaded input data, reference data, and scoring programs. Trace the dump generation entry point and verify how archive filenames are selected when multiple tasks use the same files. Done means the exported archive contains no duplicate filenames and includes only files belonging to the selected project.