codalab / codalab/codabench

CircleCI tests improvement

Open
#2,023 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement Post-it
Dominant language
Python
Stars
176
Forks
74
Avg merge
5d 3h
Merged PRs (30d)
21

Description

E2E tests (#2042)

  • E2E tests are failing on Django 4.2.0 pull request (#1959). We're remove it them but need to add them again.

  • E2E tests are randomly failing (~ 1 time out of 4).

Podman test

  • The tests only run Docker, not Podman (#1871). We need to add this.

More E2E tests

Add to the E2E tests:

  • A Codabench results submission (#2302)
  • Another Codabench code submission, typically Mini-AutoML (#2302)
  • A failing submission (#2302)
  • Several submissions made to the same benchmark
  • Multi-task tests are disabled on CI
  • Test submission on custom queue
  • All tests on codalab/codalab-legacy:py312? It is much lighter

E2E test files:
https://github.com/codalab/codabench/tree/develop/tests/test_files/competitions
https://github.com/codalab/codabench/tree/develop/tests/test_files/submissions

The test:
https://github.com/codalab/codabench/blob/develop/tests/test_submission.py

The templates:
https://github.com/codalab/competition-examples/

Improve E2E

  • Wait until Finished or Failed, and then test if Finished. (#2302)

This wait we do not wait in case of failure, and we have more informative tests.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/test_submission.py and the fixtures in tests/test_files/competitions and tests/test_files/submissions, then review the linked competition examples. Run the existing E2E tests to understand their current behavior. Done means the remaining listed submission, multi-task, custom-queue, image, and Podman test cases are covered and reliable in CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, docker, python
Domain
ci-cd, devops, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.