CircleCI tests improvement
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 176
- Forks
- 74
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 21
Description
E2E tests (#2042)
-
E2E tests are failing on Django 4.2.0 pull request (#1959). We're remove it them but need to add them again.
-
E2E tests are randomly failing (~ 1 time out of 4).
Podman test
- The tests only run Docker, not Podman (#1871). We need to add this.
More E2E tests
Add to the E2E tests:
- A Codabench results submission (#2302)
- Another Codabench code submission, typically Mini-AutoML (#2302)
- A failing submission (#2302)
- Several submissions made to the same benchmark
- Multi-task tests are disabled on CI
- Test submission on custom queue
- All tests on
codalab/codalab-legacy:py312? It is much lighter
E2E test files:
https://github.com/codalab/codabench/tree/develop/tests/test_files/competitions
https://github.com/codalab/codabench/tree/develop/tests/test_files/submissions
The test:
https://github.com/codalab/codabench/blob/develop/tests/test_submission.py
The templates:
https://github.com/codalab/competition-examples/
Improve E2E
- Wait until
FinishedorFailed, and then test ifFinished. (#2302)
This wait we do not wait in case of failure, and we have more informative tests.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/test_submission.py and the fixtures in tests/test_files/competitions and tests/test_files/submissions, then review the linked competition examples. Run the existing E2E tests to understand their current behavior. Done means the remaining listed submission, multi-task, custom-queue, image, and Podman test cases are covered and reliable in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- django, docker, python
- Domain
- ci-cd, devops, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100