codalab / codalab/codabench

More control over submissions for organizers

Open
#2,249 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement P1
Dominant language
Python
Stars
176
Forks
74
Avg merge
5d 3h
Merged PRs (30d)
21

Description

Re-run non-finished and non-failed submissions

  • Make it possible for organizers to re-run submissions in any status

It is not currently possible to re-run submissions that are stuck in a non-finished or non-failed state. Since submissions are sometimes stuck in those state, organizers might want to re-run all of them until they work correctly, which is currently not possible without first stopping the submission or deleting it entirely.

Delete soft-deleted submissions

  • Make it possible for organizers to remove soft-deleted submissions (#2422)

If a participant soft-deletes their submission, the organizer can not delete it anymore, which means that the quota can not be freed by the organizer anymore. This can become a blocking point and should probably be fixed.

Note: We do not want soft-deletion to decrease the quota to avoid abuses.

[Request] Competition option to prevent participants from removing submissions

Do we want to add this organizer option?

  • In one hand it can be useful for organizers
  • On the other hand it constraining to participants wanting to cleanup their storage quota

Note: submissions that are on the leaderboard can't be removed by participants.

Limit parallel submissions

  • Option to limit the number of parallel submissions (typically to 1)
    or
  • Option to add a submission cooldown (time before being able to submit again)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue contains several separate submission-management requests, with soft-deleted submission removal already marked complete and no files, tests, or entry points named. Start by clarifying which unchecked requirement remains in scope and its desired organizer behavior; done should be defined separately for rerunning submissions, limiting parallel submissions or cooldowns, and any participant-removal option.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.