Add CI to ensure all `requires_` tests are run at least once
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.2k
- Forks
- 1.4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 14
Description
What is your issue?
I was looking at https://github.com/pydata/xarray/issues/11251 and realised that we have tests marked as requires_cupy yet in CI have no environment providing cupy meaning these tests are (I assume) always skipped.
I think that all tests that are skipped due to a requires_ condition should either:
- pass once in one of the environments, or
- be skipped for another reason that isn't a
requires_reason
I think this can be achieved by using the existing pytest-reportlog extension, saving the resulting files, and then having a job afterwards that combines all these artifacts together again to ensure that these conditions are held. This will have informative error messaging to mention which test is never run in CI.
This will change our testing graph structure
flowchart LR
Lock --> T1["Test (env 1)"]
Lock --> T2["Test (env 2)"]
Lock --> T3["Test (env ...)"]
T1 --> c["Combine results and report status"]
T2 --> c
T3 --> c
which also means we can have a single required check on "Combine results and report status"
Other concerns:
- Can we use Pixi to define environments that only work on certain platforms? (e.g., cupy isn't available on Mac, but we shouldn't drop mac from the
pixi.toml:workspace.platforms - Are there certain things that we want to test against that aren't possible due to GHA not having compatible runners?
cc @keewis (since I've mainly worked with you on ci tooling)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing CI workflow configuration and pixi.toml, then review how pytest-reportlog produces and saves report files across environments. Trace how artifacts could be combined in a later job while preserving platform coverage. Done means CI identifies every requires_ condition that never passes or is skipped for another recorded reason, with informative failure output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100