RFE: Evaluate NixOS tests on each PR
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 323
- Forks
- 171
- PR merge metrics
- No merged PRs in 30d
Description
This weekend was tough, we've broken nixos tests eval several times (thanks @delroth for debugging those).
https://github.com/NixOS/nixpkgs/pull/55061#issuecomment-460047536
https://github.com/NixOS/nixpkgs/pull/55106
https://github.com/NixOS/nixpkgs/pull/55157
Something like
$ nix-instantiate ./. -A nixosTests >/dev/null
would have caught these issues. However the problem with above command is that:
- some tests use unfree software (
vm-test-run-bcachefs). I recall that hydra doesn't build unfree packages, does it ever build tests, which use unfree packages? Should ofborg behave the same? - some tests use insecure software:
error: Package ‘python2.7-Django-1.8.19’ in /home/danbst/dev/nixpkgs/pkgs/development/python-modules/django/1_8.nix:25 is marked as insecure, refusing to evaluate.
- tests form a tree, so some recursive walk is required. I came up with
let
# all-packages.nix
nixosTestsRecursive =
let f = n: v: builtins.trace n lib.flatten (
if builtins.typeOf v == "set" && (v.type or "" != "derivation")
then lib.attrValues (lib.mapAttrs f v)
else [ v ]
);
in lib.foldl' (acc: x: acc // { ${x.name} = x; }) {} (f "" nixosTests);
- instantiate is slow as heck. Maybe https://github.com/NixOS/nix/issues/2652 can help?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the nixosTests definition and the all-packages.nix traversal shown in the issue, then inspect how ofborg evaluates checks for each pull request. Use nix-instantiate on nixosTests as a baseline. Done means pull requests evaluate the relevant NixOS tests recursively without being blocked by the documented unfree, insecure, or performance cases.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100