codalab / codalab/codabench

Multi-task not working with Force Last submission rule

Open
#2,211 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug
Dominant language
Python
Stars
176
Forks
74
Avg merge
5d 3h
Merged PRs (30d)
21

Description

When you have more than one task and leaderboard rule is Force Last or Force Last Multiple:

  • You see N/a because one of the child submission is on the leaderboard and the other one shows N/a.
  • Also, on "My Submissions" tab, no submission is marked as "on the leaderboard". Failed submissions are marked as Finished.

We should NOT show the child submission on the leaderboard. Only the parent submission should be shown with scores from child submissions.

It looks like when the rule is applied, it is applied to submissions regardless of if they are parent or child.

Image Image

Reported in:

  • #1570
  • #1540

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the multi-task case with the Force Last and Force Last Multiple rules, then trace where the rule is applied to parent and child submissions. Done means only the parent appears on the leaderboard with the child scores, and My Submissions correctly marks leaderboard and failed statuses.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.