Tracking issue for auditing and possibly eliminating difference between PR CI and Auto CI jobs

Currently, rust-lang/rust PR jobs are not a subset of Auto jobs in the strict sense. That is, a defined PR job does not imply it is run with the exact same configuration in Auto CI.

Why might this be a problem?

When there are CI jobs (or job configurations) that are PR-only, and not also run as part of Auto jobs, it's possible to have master end up in a state where all subsequent PR jobs will be red, even in unrelated PRs. See for instance #144183 and discussions in #t-infra > Spellcheck workflow now fails on all PRs (tree bad?).

Potentially ideal CI invariant

Ideally, we may want to enforce the CI invariant that PR CI jobs are a subset of Full CI jobs in the strict-sense, with no carve-outs (other than fail-fast or not).

• See #144244 where a weaker version of the subset relationship with env and continue_on_error carve-outs is enforced. However, env being excluded is quite substantial, because env vars can substantially enforce what gets run.

Kinds of differences

[!WARNING]

This listing is non-exhaustive. Please feel free to add more known differences.

• x86_64-gnu-tools: gets DEPLOY_TOOLSTATES_JSON: toolstates-linux.json set only in Auto environment.
  • Auto: https://github.com/rust-lang/rust/blob/67819923ac8ea353aaa775303f4c3aacbf41d010/src/ci/github-actions/jobs.yml#L440-L443
  • PR: https://github.com/rust-lang/rust/blob/67819923ac8ea353aaa775303f4c3aacbf41d010/src/ci/github-actions/jobs.yml#L158-L159

Implementation history

• https://github.com/rust-lang/rust/pull/144244

Non-blocking jobs

It was discussed that we should use/implement toolstate or some other mechanism for experimental jobs, and not expose contributors to failing PR CI if said job does break.

Start with src/ci/github-actions/jobs.yml, comparing the PR and Auto job configurations around the cited x86_64-gnu-tools entries and DEPLOY_TOOLSTATES_JSON setting. Review PR #144244 and the listed issue and discussion for existing subset checks; done means auditing the known differences and defining or implementing the strict PR-is-subset-of-Auto invariant without unlisted carve-outs.