codalab / codalab/codabench

[COCO Segmentation] test-dev 2017 evaluation sever unavailable and Codabench do not have COCO benchmark

Open
#2,352 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Competition-specific Discussion
Dominant language
Python
Stars
176
Forks
74
Avg merge
5d 3h
Merged PRs (30d)
21

Description

Hi COCO team,

This is related to cocodataset/cocoapi#691.

I'm facing the same issue for the instance segmentation (mask) task specifically. The evaluation server at https://codalab.lisn.upsaclay.fr/competitions/7383 now returns "Submissions have been disabled by admins" when attempting to submit.

To summarize the current state:

  • competitions.codalab.org - fully suspended
  • codalab.lisn.upsaclay.fr/competitions/7383 (segm) - submissions disabled
  • codalab.lisn.upsaclay.fr/competitions/7384 (bbox) - submissions disabled
  • CodaBench - no active COCO competition found

This makes it impossible to report test-dev2017 results for publication, which is a significant blocker for the research community.

Could the COCO team (@cocodataset) provide any update on:

  1. Whether there is a currently active server for test-dev2017 evaluation?
  2. If a migration to CodaBench or another platform is planned, what is the expected timeline?

Thank you for your work on maintaining this benchmark.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or code entry points are identified. Review the linked cocodataset/cocoapi#691 issue and the listed CodaLab competition URLs, then determine whether an active COCO test-dev2017 evaluation service or migration plan exists; done means a maintainer-provided status or decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.