NixOS / NixOS/ofborg

RFE: Evaluate NixOS tests on each PR

Open
#315 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
323
Forks
171
PR merge metrics
No merged PRs in 30d

Description

This weekend was tough, we've broken nixos tests eval several times (thanks @delroth for debugging those).

https://github.com/NixOS/nixpkgs/pull/55061#issuecomment-460047536
https://github.com/NixOS/nixpkgs/pull/55106
https://github.com/NixOS/nixpkgs/pull/55157

Something like

$ nix-instantiate ./. -A nixosTests >/dev/null

would have caught these issues. However the problem with above command is that:

  • some tests use unfree software (vm-test-run-bcachefs). I recall that hydra doesn't build unfree packages, does it ever build tests, which use unfree packages? Should ofborg behave the same?
  • some tests use insecure software:
error: Package ‘python2.7-Django-1.8.19’ in /home/danbst/dev/nixpkgs/pkgs/development/python-modules/django/1_8.nix:25 is marked as insecure, refusing to evaluate.
  • tests form a tree, so some recursive walk is required. I came up with
let
# all-packages.nix

  nixosTestsRecursive =
    let f = n: v: builtins.trace n lib.flatten (
          if builtins.typeOf v == "set" && (v.type or "" != "derivation")
            then lib.attrValues (lib.mapAttrs f v)
            else [ v ]
        );
  in lib.foldl' (acc: x: acc // { ${x.name} = x; }) {} (f "" nixosTests);

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the nixosTests definition and the all-packages.nix traversal shown in the issue, then inspect how ofborg evaluates checks for each pull request. Use nix-instantiate on nixosTests as a baseline. Done means pull requests evaluate the relevant NixOS tests recursively without being blocked by the documented unfree, insecure, or performance cases.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, ci-cd, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.