CI Reliability 2022-12-22

Open
#465 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
20/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
git, node.js, shell

Research direction

Start with the generated CI reliability report and the linked Jenkins examples for the Git, compiler, terminal-size, JUnit, and JavaScript test failures. Compare the affected jobs, platforms, and failure logs; done means the recurring failures are diagnosed and the relevant CI or test issues are addressed so the listed jobs no longer fail intermittently.

Written by the indexing model from the issue text.

Description

Failures in node-test-pull-request/48589 to node-test-pull-request/48686 that failed more than 2 PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2022-12-22 00:13 3 5 17 2 73 5.26%
CCTest Failure
Git Failure
Reason ERROR: Error fetching remote repo 'origin'
Type GIT_FAILURE
Failed PR 15 (https://github.com/nodejs/node/pull/45877/, https://github.com/nodejs/node/pull/45882/, https://github.com/nodejs/node/pull/45258/, https://github.com/nodejs/node/pull/45793/, https://github.com/nodejs/node/pull/45547/, https://github.com/nodejs/node/pull/45876/, https://github.com/nodejs/node/pull/45857/, https://github.com/nodejs/node/pull/45895/, https://github.com/nodejs/node/pull/45886/, https://github.com/nodejs/node/pull/45885/, https://github.com/nodejs/node/pull/45887/, https://github.com/nodejs/node/pull/45913/, https://github.com/nodejs/node/pull/45847/, https://github.com/nodejs/node/pull/45401/, https://github.com/nodejs/node/pull/45803/)
Appeared test-nearform-macos10.15-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/48590/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48624/
Example
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git@github.com:nodejs/node.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:1003)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1244)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1308)
	at hudson.scm.SCM.checkout(SCM.java:540)

Build Failure
Reason ERROR: No acceptable C compiler found!
Type BUILD_FAILURE
Failed PR 18 (https://github.com/nodejs/node/pull/45877/, https://github.com/nodejs/node/pull/45258/, https://github.com/nodejs/node/pull/45793/, https://github.com/nodejs/node/pull/45547/, https://github.com/nodejs/node/pull/45876/, https://github.com/nodejs/node/pull/45895/, https://github.com/nodejs/node/pull/45869/, https://github.com/nodejs/node/pull/45885/, https://github.com/nodejs/node/pull/45913/, https://github.com/nodejs/node/pull/45401/, https://github.com/nodejs/node/pull/45803/, https://github.com/nodejs/node/pull/45857/, https://github.com/nodejs/node/pull/45847/, https://github.com/nodejs/node/pull/45914/, https://github.com/nodejs/node/pull/45887/, https://github.com/nodejs/node/pull/45882/, https://github.com/nodejs/node/pull/45886/, https://github.com/nodejs/node/pull/45936/)
Appeared test-orka-macos10.14-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/48590/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48682/
Example
ERROR: No acceptable C compiler found!

       Please make sure you have a C compiler installed on your system and/or
       consider adjusting the CC environment variable if you installed
       it in a non-standard prefix.
make: *** [build-ci] Error 1

Reason Error: getWindowSize EINVAL
Type BUILD_FAILURE
Failed PR 15 (https://github.com/nodejs/node/pull/45887/, https://github.com/nodejs/node/pull/45876/, https://github.com/nodejs/node/pull/45869/, https://github.com/nodejs/node/pull/45871/, https://github.com/nodejs/node/pull/45258/, https://github.com/nodejs/node/pull/45847/, https://github.com/nodejs/node/pull/45908/, https://github.com/nodejs/node/pull/45750/, https://github.com/nodejs/node/pull/44066/, https://github.com/nodejs/node/pull/41439/, https://github.com/nodejs/node/pull/45930/, https://github.com/nodejs/node/pull/45826/, https://github.com/nodejs/node/pull/45914/, https://github.com/nodejs/node/pull/45936/, https://github.com/nodejs/node/pull/45912/)
Appeared test-equinix_mnx-smartos18-x64-4, test-equinix_mnx-smartos20-x64-4, test-equinix_mnx-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/48598/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48685/
Example
Error: getWindowSize EINVAL
outline =     at WriteStream._refreshSize (tty.js:133:31)
outline =     at process.<anonymous> (internal/bootstrap/switches/is_main_thread.js:127:41)
outline =     at process.emit (events.js:400:28)
outline = Emitted 'error' event on WriteStream instance at:
outline =     at WriteStream._refreshSize (tty.js:133:10)

Reason ERROR: Step �Publish JUnit test result report� failed: No test report files were found. Configuration error?
Type BUILD_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/45869/, https://github.com/nodejs/node/pull/45803/, https://github.com/nodejs/node/pull/45793/, https://github.com/nodejs/node/pull/45547/, https://github.com/nodejs/node/pull/45876/)
Appeared test-nearform_arm-win10_vs2019-arm64-1, test-nearform_arm-win10_vs2019-arm64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/48589/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48606/
Example
ERROR: Step �Publish JUnit test result report� failed: No test report files were found. Configuration error?
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

Reason ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45258/, https://github.com/nodejs/node/pull/41439/)
Appeared test-osuosl-aix72-ppc64_be-3, test-equinix_mnx-smartos20-x64-3, test-osuosl-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/48601/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48672/
Example
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Sending e-mails to: michael_dawson@ca.ibm.com vieuxtech@gmail.com ashley.cripps@ibm.com
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

Jenkins Failure
JSTest Failure
Reason pummel/test-fs-readfile-tostring-fail
Type JS_TEST_FAILURE
Failed PR 11 (https://github.com/nodejs/node/pull/45847/, https://github.com/nodejs/node/pull/45826/, https://github.com/nodejs/node/pull/45908/, https://github.com/nodejs/node/pull/45886/, https://github.com/nodejs/node/pull/45887/, https://github.com/nodejs/node/pull/45912/, https://github.com/nodejs/node/pull/45885/, https://github.com/nodejs/node/pull/45258/, https://github.com/nodejs/node/pull/45878/, https://github.com/nodejs/node/pull/45914/, https://github.com/nodejs/node/pull/45923/)
Appeared test-digitalocean-freebsd12-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/48626/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48658/
Example
not ok 3451 pummel/test-fs-readfile-tostring-fail
  ---
  duration_ms: 50.674
  severity: fail
  exitcode: 7
  stack: |-
    /usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd12-x64/test/pummel/test-fs-readfile-tostring-fail.js:73
      throw err;
      ^
    
    AssertionError [ERR_ASSERTION]: 275906560 bytes written (expected 539555154 bytes).
        at WriteStream.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd12-x64/test/pummel/test-fs-readfile-tostring-fail.js:37:10)
        at WriteStream.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd12-x64/test/common/index.js:448:15)
        at WriteStream.emit (node:events:519:28)
        at finish (node:internal/streams/writable:750:10)
        at finishMaybe (node:internal/streams/writable:735:9)
        at afterWrite (node:internal/streams/writable:507:3)
        at onwrite (node:internal/streams/writable:480:7)
        at node:internal/fs/streams:443:5
        at FSReqCall...

Reason parallel/test-child-process-cwd
Type JS_TEST_FAILURE
Failed PR 10 (https://github.com/nodejs/node/pull/45865/, https://github.com/nodejs/node/pull/45851/, https://github.com/nodejs/node/pull/45869/, https://github.com/nodejs/node/pull/45890/, https://github.com/nodejs/node/pull/45803/, https://github.com/nodejs/node/pull/45258/, https://github.com/nodejs/node/pull/44004/, https://github.com/nodejs/node/pull/45750/, https://github.com/nodejs/node/pull/41439/, https://github.com/nodejs/node/pull/45923/)
Appeared test-nearform_arm-win10_vs2019-arm64-1, test-nearform_arm-win10_vs2019-arm64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/48592/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48686/
Example
not ok 102 parallel/test-child-process-cwd
  ---
  duration_ms: 0.430
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:124
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    + actual - expected
    
    + 'C:\\Windows'
    - 'C:\\WINDOWS'
            ^
        at ChildProcess.<anonymous> (c:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-child-process-cwd.js:56:26)
        at ChildProcess.<anonymous> (c:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:448:15)
        at ChildProcess.emit (node:events:519:28)
        at maybeClose (node:internal/child_process:1098:16)
        at ChildProcess._handle.onexit (node:internal/child_process:304:5) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 'C:\\Windows',
      expected: 'C:\\WINDOWS',
      operator: 'strictEqual'
    }
    
    Node.js v20.0.0-pre
  ...


Reason parallel/test-child-process-exec-cwd
Type JS_TEST_FAILURE
Failed PR 10 (https://github.com/nodejs/node/pull/45865/, https://github.com/nodejs/node/pull/45851/, https://github.com/nodejs/node/pull/45869/, https://github.com/nodejs/node/pull/45890/, https://github.com/nodejs/node/pull/45803/, https://github.com/nodejs/node/pull/45258/, https://github.com/nodejs/node/pull/44004/, https://github.com/nodejs/node/pull/45750/, https://github.com/nodejs/node/pull/41439/, https://github.com/nodejs/node/pull/45923/)
Appeared test-nearform_arm-win10_vs2019-arm64-1, test-nearform_arm-win10_vs2019-arm64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/48592/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48686/
Example
not ok 103 parallel/test-child-process-exec-cwd
  ---
  duration_ms: 0.498
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:400
        throw err;
        ^
    
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(stdout.startsWith(dir))
    
        at c:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-child-process-exec-cwd.js:38:3
        at c:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:413:17
        at c:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:448:15
        at ChildProcess.exithandler (node:child_process:411:7)
        at ChildProcess.emit (node:events:519:28)
        at maybeClose (node:internal/child_process:1098:16)
        at ChildProcess._handle.onexit (node:internal/child_process:304:5) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '=='
    }
    
    Node.js v20.0.0-pre
  ...


Reason parallel/test-fs-watch-recursive
Type JS_TEST_FAILURE
Failed PR 8 (https://github.com/nodejs/node/pull/45882/, https://github.com/nodejs/node/pull/45869/, https://github.com/nodejs/node/pull/45885/, https://github.com/nodejs/node/pull/45847/, https://github.com/nodejs/node/pull/45401/, https://github.com/nodejs/node/pull/45857/, https://github.com/nodejs/node/pull/45914/, https://github.com/nodejs/node/pull/45826/)
Appeared test-equinix-rhel8_container-arm64-2, test-orka-macos10.15-x64-2, test-orka-macos11-x64-2, test-osuosl-rhel8-ppc64_le-4, test-orka-macos11-x64-1, test-equinix-ubuntu1804_container-arm64-2, test-nearform-macos11.0-arm64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/48597/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48638/
Example
not ok 1265 parallel/test-fs-watch-recursive
  ---
  duration_ms: 5.601
  severity: fail
  exitcode: 1
  stack: |-
    (node:838111) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
    (Use `node --trace-warnings ...` to show where the warning was created)
    node:internal/process/promises:289
                triggerUncaughtException(err, true /* fromPromise */);
                ^
    
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(!process._getActiveHandles().some((handle) => handle.constructor.name === 'StatWatcher'))
    
        at FSWatcher.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/test/parallel/test-fs-watch-recursive.js:264:5) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '=='
    }
    
    Node.js v20.0.0-pre
  ...


Reason parallel/test-runner-reporters
Type JS_TEST_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/45793/, https://github.com/nodejs/node/pull/45876/, https://github.com/nodejs/node/pull/45886/, https://github.com/nodejs/node/pull/45803/, https://github.com/nodejs/node/pull/45887/, https://github.com/nodejs/node/pull/44066/, https://github.com/nodejs/node/pull/45923/)
Appeared test-digitalocean-freebsd12-x64-2, test-ibm-rhel8-s390x-4, test-azure_msft-win10_vs2019-x64-1, test-rackspace-win2012r2_vs2019-x64-4, test-osuosl-rhel8-ppc64_le-2, test-digitalocean-freebsd12-x64-1, test-ibm-rhel8-s390x-2, test-osuosl-rhel8-ppc64_le-4
First CI https://ci.nodejs.org/job/node-test-pull-request/48602/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48686/
Example
not ok 2165 parallel/test-runner-reporters
  ---
  duration_ms: 3.616
  severity: fail
  exitcode: 1
  stack: |-
    TAP version 13
    # Subtest: node:test reporters
        # Subtest: should default to outputing TAP to stdout
        ok 1 - should default to outputing TAP to stdout
          ---
          duration_ms: 3340.490923
          ...


Reason pummel/test-net-throttle
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/45258/, https://github.com/nodejs/node/pull/45876/, https://github.com/nodejs/node/pull/45847/, https://github.com/nodejs/node/pull/45878/)
Appeared test-digitalocean-freebsd12-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/48601/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48631/
Example
not ok 3472 pummel/test-net-throttle
  ---
  duration_ms: 0.577
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:124
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    false !== true
    
        at process.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd12-x64/test/pummel/test-net-throttle.js:87:10)
        at process.emit (node:events:531:35) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: 'strictEqual'
    }
    
    Node.js v20.0.0-pre
  ...


Reason parallel/test-watch-mode-files_watcher
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/45793/, https://github.com/nodejs/node/pull/45847/, https://github.com/nodejs/node/pull/45887/)
Appeared test-equinix-debian10_container-armv7l-2, test-orka-macos11-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/48602/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48628/
Example
not ok 3072 parallel/test-watch-mode-files_watcher
  ---
  duration_ms: 0.599
  severity: fail
  exitcode: 1
  stack: |-
    TAP version 13
    # Subtest: watch mode file watcher
        # Subtest: should watch changed files
        not ok 1 - should watch changed files
          ---
          duration_ms: 4.253458
          failureType: 'testCodeFailure'
          error: "EMFILE: too many open files, watch '/home/iojs/tmp/.tmp.3073/file1'"
          code: 'EMFILE'
          stack: |-
            FSWatcher.<computed> (node:internal/fs/watchers:244:19)
            watch (node:fs:2304:36)
            FilesWatcher.watchPath (node:internal/watch_mode/files_watcher:92:21)
            FilesWatcher.filterFile (node:internal/watch_mode/files_watcher:108:12)
            Object.<anonymous> (file:///home/iojs/build/workspace/node-test-binary-armv7l/test/parallel/test-watch-mode-files_watcher.mjs:49:13)
            ItTest.runInAsyncScope (node:async_hooks:204:9)
            ItTest.run (node:internal/test_runner/test:547:...

Reason parallel/test-heapsnapshot-near-heap-limit-by-api-in-worker
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45803/, https://github.com/nodejs/node/pull/45923/)
Appeared test-osuosl-rhel8-ppc64_le-2, test-azure_msft-win10_vs2019-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/48624/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48674/
Example
not ok 1212 parallel/test-heapsnapshot-near-heap-limit-by-api-in-worker
  ---
  duration_ms: 1.937
  severity: fail
  exitcode: 1
  stack: |-
    
    Invoked NearHeapLimitCallback, processing=false, current_limit=52428800, initial_limit=52428800
    max_young_gen_size=50331648, young_gen_size=0, old_gen_size=51542712, total_size=51542712
    Estimated available memory=9223372036749459456, estimated overhead=50331648
    Start generating Heap.20221221.122414.1983031.1.001.heapsnapshot...
    Invoked NearHeapLimitCallback, processing=true, current_limit=52428800, initial_limit=52428800
    max_young_gen_size=50331648, young_gen_size=0, old_gen_size=51542712, total_size=51542712
    Estimated available memory=9223372036749459456, estimated overhead=50331648
    Not generating snapshots in nested callback. new_limit=102760448
    1/1 snapshots taken.
    Removing the near heap limit callbackWrote snapshot to /home/iojs/node-tmp/.tmp.1211/Heap.20221221.122414.1983031.1.001.heapsnapshot
    
    <--- Last few GCs ...

Reason sequential/test-inspector-debug-end
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45857/, https://github.com/nodejs/node/pull/45886/)
Appeared test-ibm-rhel8-s390x-2, test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/48625/
Last CI https://ci.nodejs.org/job/node-test-pull-request/48629/
Example
not ok 3796 sequential/test-inspector-debug-end
  ---
  duration_ms: 120.103
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    Test there's no crash stopping server that was not started
    Test there's no crash stopping server without connecting
    [err] Debugger listening on ws://127.0.0.1:33929/98861b08-8991-4e08-a5ab-10b3fefdfc2c
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    Test there's no crash stopping server after connecting
    [test] Connecting to a child Node process
    [test] Testing /json/list
    [err] Debugger listening on ws://127.0.0.1:36661/974bcc88-5697-40b7-bc2e-a917b616e3aa
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    [err] Debugger attached.
    [err] 
  ...


Progress
  • ERROR: Error fetching remote repo 'origin' (15)
  • ERROR: No acceptable C compiler found! (18)
  • Error: getWindowSize EINVAL (15)
  • ERROR: Step �Publish JUnit test result report� failed: No test report files were found. Configuration error? (5)
  • ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? (2)
  • pummel/test-fs-readfile-tostring-fail (11)
  • parallel/test-child-process-cwd (10)
  • parallel/test-child-process-exec-cwd (10)
  • parallel/test-fs-watch-recursive (8)
  • parallel/test-runner-reporters (7)
  • pummel/test-net-throttle (4)
  • parallel/test-watch-mode-files_watcher (3)
  • parallel/test-heapsnapshot-near-heap-limit-by-api-in-worker (2)
  • sequential/test-inspector-debug-end (2)
Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/reliability

All issues in nodejs/reliability

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.