pytorch / pytorch/vision

Surface failing CI jobs that only run on main

Open
#7,374 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement module: ci
Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

On CircleCI we still have these Linux GPU tests:

https://github.com/pytorch/vision/blob/8499bc497b0b1d19f971b7389623884bcd44c62b/.circleci/config.yml#L1353-L1374

They are replaced by

https://github.com/pytorch/vision/blob/8499bc497b0b1d19f971b7389623884bcd44c62b/.github/workflows/test-linux-gpu.yml#L17-L21

Although it looks like if this just drops the GPU workflows for Python 3.9 and 3.10, one has to pay attention to the filter key in the CircleCI config: these workflows are only run on the main and the nightly branch.

I can only speak for myself here, but I very rarely have look a the CI on main, but rather operate under the assumption "if the CI in the PR is green, life is good". If the other maintainer feel similar, we are basically running these tests for nothing. Meaning, we can remove them in #7354 without reducing the "apparent coverage".

However, this of course still means that we are reducing coverage. If we can't run this workflows on PRs for monetary reasons, we need a way to surface failing jobs that only run on main. We have such a mechanism already for the dataset availability tests: https://github.com/pytorch/vision/blob/main/.github/workflows/tests-schedule.yml. We just run our tests there on a daily schedule and a bot will open an issue in case something fails like #7361. From there on, we can handle it manually, since this will hopefully not happen often.

cc @seemethere

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare the Linux GPU jobs in .circleci/config.yml at lines 1353-1374 with .github/workflows/test-linux-gpu.yml at lines 17-21, paying attention to the main/nightly filter. Then inspect .github/workflows/tests-schedule.yml and issue #7361 for the existing scheduled-test and failure-reporting pattern. Done means failures from the main-only jobs are surfaced through a scheduled workflow and follow-up issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.