codalab / codalab/codabench

MOT20 leaderboard ranks a submission with no evaluation scores as #1

Open
#2,470 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
176
Forks
74
Avg merge
5d 3h
Merged PRs (30d)
21

Description

Hello Codabench team,

I found a ranking issue on the MOT20 Challenge leaderboard for MOT20 Test set submissions.

A submission with no evaluation scores is currently ranked #1, ahead of submissions with valid results.

Current leaderboard entries

Rank 1

Participant: s*****
Date: 2026-07-22 17:58
Submission ID: 857861
Method name: adt
Detection type: Public Detections
Tracker type: Online Tracker
HOTA: n/a
MOTA: n/a
IDF1: n/a
All other evaluation metrics: n/a

Expected behavior

Submissions without valid evaluation results should not be ranked above successfully evaluated submissions. They should either:

be excluded from the leaderboard ranking,
be marked as failed or pending, or
appear below submissions with valid scores.

Actual behavior

Submission 857861, which has n/a for all evaluation metrics, is ranked first on the leaderboard.

Could you please check the leaderboard sorting and handling of submissions with missing evaluation scores?

Thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the MOT20 leaderboard and submission 857861, then trace the leaderboard sorting and handling of submissions whose evaluation metrics are all n/a. Reproduce the ranking behavior and verify that submissions without valid scores are excluded, marked failed or pending, or placed below evaluated submissions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.