Surface failing CI jobs that only run on main
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 7.3k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 13
Description
On CircleCI we still have these Linux GPU tests:
They are replaced by
Although it looks like if this just drops the GPU workflows for Python 3.9 and 3.10, one has to pay attention to the filter key in the CircleCI config: these workflows are only run on the main and the nightly branch.
I can only speak for myself here, but I very rarely have look a the CI on main, but rather operate under the assumption "if the CI in the PR is green, life is good". If the other maintainer feel similar, we are basically running these tests for nothing. Meaning, we can remove them in #7354 without reducing the "apparent coverage".
However, this of course still means that we are reducing coverage. If we can't run this workflows on PRs for monetary reasons, we need a way to surface failing jobs that only run on main. We have such a mechanism already for the dataset availability tests: https://github.com/pytorch/vision/blob/main/.github/workflows/tests-schedule.yml. We just run our tests there on a daily schedule and a bot will open an issue in case something fails like #7361. From there on, we can handle it manually, since this will hopefully not happen often.
cc @seemethere
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the Linux GPU jobs in .circleci/config.yml at lines 1353-1374 with .github/workflows/test-linux-gpu.yml at lines 17-21, paying attention to the main/nightly filter. Then inspect .github/workflows/tests-schedule.yml and issue #7361 for the existing scheduled-test and failure-reporting pattern. Done means failures from the main-only jobs are surfaced through a scheduled workflow and follow-up issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100