run_tests v1.0.0 can't express QA-only-for-GROUP==QA (QA excluded from "All"); blocks single-package rollout
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 1
- Forks
- 1
- Avg merge
- 10h 1m
- Merged PRs (30d)
- 13
Description
Summary
run_tests v1.0.0 cannot reproduce the very common SciML pattern where the QA group runs only for GROUP=="QA" and is excluded from "All" (the default local ]test path). This blocks the SciMLTesting rollout for every single-package repo that follows that pattern.
Affected repos found during the rollout fan-out (all four run QA only under GROUP=="QA", never under "All"):
- GlobalSensitivity.jl
- HighDimPDE.jl
- IRKGaussLegendre.jl
- Integrals.jl
These repos' hand-written test/runtests.jl all have a standalone if GROUP == "QA" ... end branch plus an if GROUP == "All" || GROUP == "Core" (or bare else) branch that does not include QA. So under a plain local ]test (GROUP unset → "All"), QA does not run today.
The gap
In v1.0.0 there are exactly two ways to wire QA, and neither preserves "QA only for GROUP==QA, never under All":
-
qa=kwarg — runs QA forGROUP=="QA"and under"All"(src line 494:qa === nothing || _run_group_spec(...)inside thegroup == "All"branch). Using this changes behavior:]testwould newly run QA. -
groups["QA"]with a declaredenv— correctly excluded from"All"(agroupsentry withenv !== nothingis skipped under All), but unreachable whenGROUP=="QA": theelseif group == "QA"branch (line 499) short-circuits before theelseif haskey(group_table, group)branch (line 504) and throwsArgumentError("run_tests: GROUP=\"QA\" was requested but no \qa` body was provided")`.
Reproduction
using SciMLTesting
# (1) qa= kwarg: QA runs under "All" (behavior change vs. the repos above)
ran = String[]
withenv("GROUP" => "All") do
run_tests(; core = () -> push!(ran, "core"),
qa = (; env = qa_env_dir, body = () -> push!(ran, "QA")))
end
@assert ran == ["core", "QA"] # QA leaked into All
# (2) groups["QA"] with env: unreachable for GROUP=="QA"
withenv("GROUP" => "QA") do
run_tests(; core = () -> nothing,
groups = Dict("QA" => (; env = qa_env_dir, body = () -> nothing)))
end
# ERROR: ArgumentError: run_tests: GROUP="QA" was requested but no `qa` body was provided
Suggested fix
Allow a QA body that is selectable by GROUP=="QA" yet excluded from "All". Options:
- Add an opt-out flag to the
qa=spec, e.g.qa = (; env, body, in_all = false), so thegroup == "All"branch skips it whenin_all == false. Default could staytruefor back-compat. - OR reorder/extend the
GROUP=="QA"branch so agroups["QA"]entry is honored (i.e. checkgroup_tablefor"QA"before erroring onqa === nothing). This makes the documented "declared-env group is excluded from All" mechanism work uniformly for QA too.
Either fix unblocks the four repos above (and the broader class). Until then those conversions are correctly held as blocked to avoid a behavior change to the local ]test path.
Verified against SciMLTesting v1.0.0 on Julia 1.11.9.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the run_tests implementation around src lines 494-504 and inspect how qa and groups entries are selected for GROUP="All" and GROUP="QA". Use the two reproductions in the issue as regression cases; done means a QA body can run for GROUP="QA" while remaining excluded from GROUP="All", without changing existing defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100