SciML / SciML/SciMLTesting.jl

run_tests v1.0.0 can't express QA-only-for-GROUP==QA (QA excluded from "All"); blocks single-package rollout

Open
#5 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
1
Forks
1
Avg merge
10h 1m
Merged PRs (30d)
13

Description

Summary

run_tests v1.0.0 cannot reproduce the very common SciML pattern where the QA group runs only for GROUP=="QA" and is excluded from "All" (the default local ]test path). This blocks the SciMLTesting rollout for every single-package repo that follows that pattern.

Affected repos found during the rollout fan-out (all four run QA only under GROUP=="QA", never under "All"):

  • GlobalSensitivity.jl
  • HighDimPDE.jl
  • IRKGaussLegendre.jl
  • Integrals.jl

These repos' hand-written test/runtests.jl all have a standalone if GROUP == "QA" ... end branch plus an if GROUP == "All" || GROUP == "Core" (or bare else) branch that does not include QA. So under a plain local ]test (GROUP unset → "All"), QA does not run today.

The gap

In v1.0.0 there are exactly two ways to wire QA, and neither preserves "QA only for GROUP==QA, never under All":

  1. qa= kwarg — runs QA for GROUP=="QA" and under "All" (src line 494: qa === nothing || _run_group_spec(...) inside the group == "All" branch). Using this changes behavior: ]test would newly run QA.

  2. groups["QA"] with a declared env — correctly excluded from "All" (a groups entry with env !== nothing is skipped under All), but unreachable when GROUP=="QA": the elseif group == "QA" branch (line 499) short-circuits before the elseif haskey(group_table, group) branch (line 504) and throws ArgumentError("run_tests: GROUP=\"QA\" was requested but no \qa` body was provided")`.

Reproduction
using SciMLTesting

# (1) qa= kwarg: QA runs under "All" (behavior change vs. the repos above)
ran = String[]
withenv("GROUP" => "All") do
    run_tests(; core = () -> push!(ran, "core"),
              qa = (; env = qa_env_dir, body = () -> push!(ran, "QA")))
end
@assert ran == ["core", "QA"]   # QA leaked into All

# (2) groups["QA"] with env: unreachable for GROUP=="QA"
withenv("GROUP" => "QA") do
    run_tests(; core = () -> nothing,
              groups = Dict("QA" => (; env = qa_env_dir, body = () -> nothing)))
end
# ERROR: ArgumentError: run_tests: GROUP="QA" was requested but no `qa` body was provided

Suggested fix

Allow a QA body that is selectable by GROUP=="QA" yet excluded from "All". Options:

  • Add an opt-out flag to the qa= spec, e.g. qa = (; env, body, in_all = false), so the group == "All" branch skips it when in_all == false. Default could stay true for back-compat.
  • OR reorder/extend the GROUP=="QA" branch so a groups["QA"] entry is honored (i.e. check group_table for "QA" before erroring on qa === nothing). This makes the documented "declared-env group is excluded from All" mechanism work uniformly for QA too.

Either fix unblocks the four repos above (and the broader class). Until then those conversions are correctly held as blocked to avoid a behavior change to the local ]test path.

Verified against SciMLTesting v1.0.0 on Julia 1.11.9.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the run_tests implementation around src lines 494-504 and inspect how qa and groups entries are selected for GROUP="All" and GROUP="QA". Use the two reproductions in the issue as regression cases; done means a QA body can run for GROUP="QA" while remaining excluded from GROUP="All", without changing existing defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.