PR check doesn't fail even if all `passthru.tests` fail
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 322
- Forks
- 171
- PR merge metrics
- No merged PRs in 30d
Description
Currently, ofborg marks failing passthru.tests as neutral (grey box), so a ofborg pkg check can succeed even if the tests fail on all platforms.
This recently happend in package update PR https://github.com/NixOS/nixpkgs/pull/206835, which broke the corresponding NixOS module (later fixed here).
The ofborg check was green, and the faulty PR got merged.
Here's a simple fix that would be sufficient to detect most failures:
The PR CI check should succeed only if each test in passthru.tests succeeds on at least one platform.
(A thorough, but complex alternative is to define which platforms are supported for a test in nixpkgs. ofborg should then fail if a test on a supported platform fails.)
cc @NobbZ, @SuperSandro2000
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named in the issue. Start by tracing how ofborg evaluates passthru.tests for the PR CI check, then inspect existing check-status tests or fixtures. Done means the check fails when any test fails on every platform, while passing when each test succeeds on at least one platform.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ci-cd, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100