NixOS / NixOS/hydra

Corner case in hydra-eval-jobset existing build scheduled check?

Open
#697 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PLpgSQL
Stars
1.6k
Forks
358
Avg merge
20h 30m
Merged PRs (30d)
27

Description

I pushed a derivation for mfixgui and mfixsolver that had an invalid url in a fetchurl field. The hydra build failed as expected.

( fetchurl {
    url = "https://mfix.netl.doe.gov/s3/71e94306/f00d8cf543e7a1f42c41a545baef58c4/source/mfix/mfix-${version}.tar.gz";
    sha256 = "1bnwvgmrmz9y32fsnnqb4x2ab4yh9iyg1d7f3wc2w6q9fsfnzpj4";
} )
trying https://mfix.netl.doe.gov/s3/71e94306/f00d8cf543e7a1f42c41a545baef58c4/source/mfix/mfix-19.3.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
100 34322    0 34322    0     0   9147      0 --:--:--  0:00:03 --:--:-- 84536
100   220  100   220    0     0     56      0  0:00:03  0:00:03 --:--:--    56
100 12542    0 12542    0     0   3125      0 --:--:--  0:00:04 --:--:--  3125
hash mismatch in fixed-output derivation '/nix/store/8qmh1xvg53d3la951xz710617ll9bwp7-mfix-19.3.0.tar.gz':
  wanted: sha256:1bnwvgmrmz9y32fsnnqb4x2ab4yh9iyg1d7f3wc2w6q9fsfnzpj4
  got:    sha256:1h43xxf315hwn7s3fz7xphkilgwh3m1ga7h5ysr25m60nwfh0mb1

I then amended the commit with the corrected url value and force pushed. The hydra build is continuing to fail and showing a build log with the original failed download step with the incorrect url.

( fetchurl {
    url = "https://www.sharcnet.ca/~tyson/nix/mfix-${version}.tar.gz";
    sha256 = "1bnwvgmrmz9y32fsnnqb4x2ab4yh9iyg1d7f3wc2w6q9fsfnzpj4";
} )
trying https://mfix.netl.doe.gov/s3/71e94306/f00d8cf543e7a1f42c41a545baef58c4/source/mfix/mfix-19.3.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
100 34322    0 34322    0     0   9147      0 --:--:--  0:00:03 --:--:-- 84536
100   220  100   220    0     0     56      0  0:00:03  0:00:03 --:--:--    56
100 12542    0 12542    0     0   3125      0 --:--:--  0:00:04 --:--:--  3125
hash mismatch in fixed-output derivation '/nix/store/8qmh1xvg53d3la951xz710617ll9bwp7-mfix-19.3.0.tar.gz':
  wanted: sha256:1bnwvgmrmz9y32fsnnqb4x2ab4yh9iyg1d7f3wc2w6q9fsfnzpj4
  got:    sha256:1h43xxf315hwn7s3fz7xphkilgwh3m1ga7h5ysr25m60nwfh0mb1

After much hair pulling, I eventually stepped through the hydra-eval-jobset script in perl debug mode and believe I have discovered a corner case. The checkBuild function checks for previous builds with the following search, and doesn't add a new build if it finds a match

my ($prevBuild) = $prevEval->builds->search(
    # The "project" and "jobset" constraints are                                                        
    # semantically unnecessary (because they're implied by                                              
    # the eval), but they give a factor 1000 speedup on                                                 
    # the Nixpkgs jobset with PostgreSQL.                                                               
    { project => $jobset->project->name, jobset => $jobset->name, job => $jobName,
      name => $firstOutputName, path => $firstOutputPath },
    { rows => 1, columns => ['id'], join => ['buildoutputs'] });

I think this causes problems in the case I've laid out above as the overall output path remains the same after the updated url push because the only change made was to the derivation for a fixed output input.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the hydra-eval-jobset checkBuild function and the prevEval->builds->search call shown in the report. Compare evaluations 550 and 551 with the two mfixgui.nix revisions, and verify whether the corrected fixed-output URL reuses the previous output path and skips the build; done means this case schedules the corrected build.

Written by the indexing model from the issue text.

Assessment

Tech stack
perl, postgresql
Domain
build-system, ci-cd
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.