pydata / pydata/xarray

Add CI to ensure all `requires_` tests are run at least once

Open
#11,273 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

CI enhancement topic-testing
Dominant language
Python
Stars
4.2k
Forks
1.4k
Avg merge
2d 15h
Merged PRs (30d)
14

Description

What is your issue?

I was looking at https://github.com/pydata/xarray/issues/11251 and realised that we have tests marked as requires_cupy yet in CI have no environment providing cupy meaning these tests are (I assume) always skipped.

I think that all tests that are skipped due to a requires_ condition should either:

  • pass once in one of the environments, or
  • be skipped for another reason that isn't a requires_ reason

I think this can be achieved by using the existing pytest-reportlog extension, saving the resulting files, and then having a job afterwards that combines all these artifacts together again to ensure that these conditions are held. This will have informative error messaging to mention which test is never run in CI.

This will change our testing graph structure

flowchart LR
    Lock --> T1["Test (env 1)"]
    Lock --> T2["Test (env 2)"]
    Lock --> T3["Test (env ...)"]
    T1 --> c["Combine results and report status"]
    T2 --> c
    T3 --> c

which also means we can have a single required check on "Combine results and report status"

Other concerns:

  • Can we use Pixi to define environments that only work on certain platforms? (e.g., cupy isn't available on Mac, but we shouldn't drop mac from the pixi.toml:workspace.platforms
  • Are there certain things that we want to test against that aren't possible due to GHA not having compatible runners?

cc @keewis (since I've mainly worked with you on ci tooling)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing CI workflow configuration and pixi.toml, then review how pytest-reportlog produces and saves report files across environments. Trace how artifacts could be combined in a later job while preserving platform coverage. Done means CI identifies every requires_ condition that never passes or is skipped for another recorded reason, with informative failure output.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.