Submit independently to tasks

General idea

Right now, submitting to only one task results in having n/a scores to the other tasks. It would be interesting to be able to submit seperately to different tasks and have the results merged on the same leaderboard row.

Potential problems

• In some cases, we want the exact same submission to be evaluated on the different tasks. Therefore, allowing participants to submit different models / results to the different tasks, and then merging the scores, may be problematic.

• The leaderboard definition (score names, etc.) is the same for each task.

Potential implementations

Different possible approaches to discuss:

• Using the current system of selecting tasks to which submit. Then, the submissions would be added to the leaderboard depending on the submission rule (manually by the user, or the best first, etc.). It is not really clear how to implement this without deep changes.

• Using multiple leaderboards (#1553)

• Allowing phases to be simultaneous, instead of sequential. This way, the different tasks could be separated in different phases. That may require deep changes.

Current work-around

The current work-around, used by competitions organizers, is to open separate competitions (with separate URLs) for each task.

Related issue

• #1679
• #1570
• #1334
• The meta scoring problem could be done on tasks, not on columns: #406

Start by reviewing the existing task-selection and submission flow, then read related issues #1679, #1570, #1334, #406, and the multiple-leaderboards proposal in #1553. Compare the possible approaches described here and clarify how identical submissions, leaderboard definitions, and score merging should work. Done means participants can submit tasks independently while appearing in one leaderboard row without breaking current submission rules.