CI Reliability 2022-10-01

Open
#391 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
shell
Domain
ci-cd, devops

Research direction

Start with the generated ncu-ci walk pr --stats=true report and the linked Jenkins console logs for the listed jobs, including pummel/test-fs-largefile, parallel/test-worker-heap-snapshot, and sequential/test-http-regr-gh-2928. Done requires determining actionable causes and follow-up changes for the recurring CI and test failures, but the issue does not identify files or a specific fix.

Written by the indexing model from the issue text.

Description

Failures in node-test-pull-request/46866 to node-test-pull-request/46948 that failed more than 2 PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2022-10-01 00:23 11 8 19 6 56 9.64%
Jenkins Failure
Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/44690/, https://github.com/nodejs/node/pull/44766/, https://github.com/nodejs/node/pull/44686/, https://github.com/nodejs/node/pull/44692/, https://github.com/nodejs/node/pull/44689/)
Appeared test-digitalocean-freebsd12-x64-1, test-digitalocean-freebsd12-x64-2, test-nearform-macos10.15-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46884/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46918/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 45.55.90.237/45.55.90.237:20885' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at jdk.proxy2/jdk.proxy2.$Proxy92.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207)

Build Failure
Reason Failed to trigger node-test-commit
Type BUILD_FAILURE
Failed PR 8 (https://github.com/nodejs/node/pull/44772/, https://github.com/nodejs/node/pull/44746/, https://github.com/nodejs/node/pull/44809/, https://github.com/nodejs/node/pull/44697/, https://github.com/nodejs/node/pull/44693/, https://github.com/nodejs/node/pull/44690/, https://github.com/nodejs/node/pull/44689/, https://github.com/nodejs/node/pull/44686/)
Appeared test-equinix-ubuntu2204-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/46873/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46904/
Example
Failed to trigger node-test-commit

Reason error: remote GitHub already exists.
Type BUILD_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44741/, https://github.com/nodejs/node/pull/44561/, https://github.com/nodejs/node/pull/44746/)
Appeared test-rackspace-win2012r2_vs2017-x64-1, test-rackspace-win2012r2_vs2019-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/46886/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46925/
Example
error: remote GitHub already exists.

> git fetch --prune --no-tags GitHub 

> git fetch --prune --no-tags https://github.com/nodejs/build.git +refs/heads/main:refs/remotes/jenkins_tmp

Reason Error: [WinError 5] Access is denied: 'C:\Users\Administrator\clcache\stats.txt.new' -> 'C:\Users\Administrator\clcache\stats.txt'
Type BUILD_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44677/, https://github.com/nodejs/node/pull/44692/)
Appeared test-rackspace-win2012r2_vs2019-x64-2, test-rackspace-win2012r2_vs2019-x64-5
First CI https://ci.nodejs.org/job/node-test-pull-request/46886/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46910/
Example
Error: [WinError 5] Access is denied: 'C:\\Users\\Administrator\\clcache\\stats.txt.new' -> 'C:\\Users\\Administrator\\clcache\\stats.txt'
  [1228] Failed to execute script clcache_main
  frames-tq.cc
  finalization-registry-tq.cc
  frame-arguments-tq.cc
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(523,5): error MSB6006: "clcache_main.exe" exited with code -1. [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_base_without_compiler.vcxproj]

Reason ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44746/, https://github.com/nodejs/node/pull/44731/)
Appeared test-osuosl-aix72-ppc64_be-4, test-equinix-rhel8_container-arm64-1, test-equinix-ubuntu1804_container-arm64-2, test-equinix-ubuntu2004_container-arm64-2, test-equinix-ubuntu2004_container-armv7l-1, test-joyent-smartos20-x64-3, test-equinix-ubuntu1804_sharedlibs_container-arm64-3, test-equinix-ubuntu1804_sharedlibs_container-arm64-2, test-equinix-ubuntu1804_sharedlibs_container-arm64-6
First CI https://ci.nodejs.org/job/node-test-pull-request/46932/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46936/
Example
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Sending e-mails to: michael_dawson@ca.ibm.com vieuxtech@gmail.com ashley.cripps@ibm.com
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

JSTest Failure
Reason pummel/test-fs-largefile
Type JS_TEST_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/44693/, https://github.com/nodejs/node/pull/44799/, https://github.com/nodejs/node/pull/44697/, https://github.com/nodejs/node/pull/44675/, https://github.com/nodejs/node/pull/44766/, https://github.com/nodejs/node/pull/44767/)
Appeared test-rackspace-win2012r2_vs2019-x64-4, test-nearform-macos10.15-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/46869/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46907/
Example
not ok 843 pummel/test-fs-largefile
  ---
  duration_ms: 0.943
  severity: fail
  exitcode: 1
  stack: |-
    Can't clean tmpdir: C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.844
    Files blocking: [ 'large.txt' ]
    
    node:internal/fs/utils:347
        throw err;
        ^
    
    Error: ENOSPC: no space left on device, ftruncate
        at Object.ftruncateSync (node:fs:1126:3)
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\pummel\test-fs-largefile.js:37:4)
        at Module._compile (node:internal/modules/cjs/loader:1159:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
        at Module.load (node:internal/modules/cjs/loader:1037:32)
        at Module._load (node:internal/modules/cjs/loader:878:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:86:12)
        at node:internal/main/run_main_module:23:47 {
      errno: -4055,
      syscall: 'ftruncate',
      code: 'ENOSPC'...

Reason parallel/test-worker-heap-snapshot
Type JS_TEST_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/44796/, https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44695/, https://github.com/nodejs/node/pull/44809/, https://github.com/nodejs/node/pull/44815/)
Appeared test-rackspace-win2012r2_vs2013-x64-1, test-rackspace-win2012r2_vs2019-x64-4, test-rackspace-win2012r2_vs2019-x64-6, test-rackspace-win2012r2_vs2017-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/46889/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46930/
Example
not ok 803 parallel/test-worker-heap-snapshot
  ---
  duration_ms: 1.360
  severity: fail
  exitcode: 134
  stack: |-
    FATAL ERROR: HandleScope::HandleScope Entering the V8 API without proper locking in place
  ...


Reason sequential/test-http-regr-gh-2928
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/44809/, https://github.com/nodejs/node/pull/44796/, https://github.com/nodejs/node/pull/44692/)
Appeared test-digitalocean-freebsd12-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/46912/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46939/
Example
not ok 3714 sequential/test-http-regr-gh-2928
  ---
  duration_ms: 120.87
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    .+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.
  ...


Reason sequential/test-watch-mode-inspect
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/44798/, https://github.com/nodejs/node/pull/44796/, https://github.com/nodejs/node/pull/44686/)
Appeared test-nearform-macos10.15-x64-1, test-orka-macos10.15-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46923/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46938/
Example
not ok 3791 sequential/test-watch-mode-inspect
  ---
  duration_ms: 120.147
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    TAP version 13
    (node:61962) ExperimentalWarning: The test runner is an experimental feature. This feature could change at any time
    (Use `node --trace-warnings ...` to show where the warning was created)
  ...


Reason sequential/test-worker-prof
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/44766/, https://github.com/nodejs/node/pull/44690/, https://github.com/nodejs/node/pull/44686/)
Appeared test-azure_msft-win10_vs2019-x64-2, test-azure_msft-win10_vs2019-x64-3, test-azure_msft-win10_vs2019-x64-1, test-equinix-ubuntu1804_sharedlibs_container-arm64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/46885/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46943/
Example
not ok 898 sequential/test-worker-prof
  ---
  duration_ms: 30.304
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:124
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: child exited with signal: {
      error: Error: spawnSync C:\workspace\node-test-binary-windows-js-suites\node\Release\node.exe ETIMEDOUT
          at Object.spawnSync (node:internal/child_process:1117:20)
          at spawnSync (node:child_process:857:24)
          at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\sequential\test-worker-prof.js:53:23)
          at Module._compile (node:internal/modules/cjs/loader:1159:14)
          at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
          at Module.load (node:internal/modules/cjs/loader:1037:32)
          at Module._load (node:internal/modules/cjs/loader:878:12)
          at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:86:12)
          at node:internal/main/run_mai...

Reason abort/test-addon-uv-handle-leak
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44766/, https://github.com/nodejs/node/pull/44677/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/46885/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46898/
Example
not ok 3630 abort/test-addon-uv-handle-leak
  ---
  duration_ms: 300.129
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason addons/request-interrupt/test
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44815/, https://github.com/nodejs/node/pull/44813/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/46934/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46948/
Example
not ok 3460 addons/request-interrupt/test
  ---
  duration_ms: 300.67
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-tls-hello-parser-failure
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44741/, https://github.com/nodejs/node/pull/44686/)
Appeared test-rackspace-win2012r2_vs2017-x64-3, test-osuosl-rhel8-ppc64_le-2
First CI https://ci.nodejs.org/job/node-test-pull-request/46887/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46943/
Example
not ok 699 parallel/test-tls-hello-parser-failure
  ---
  duration_ms: 0.312
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:124
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    false !== true
    
        at Socket.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-tls-hello-parser-failure.js:64:12)
        at Socket.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:446:15)
        at Socket.emit (node:events:513:28)
        at TCP.<anonymous> (node:net:298:12) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: 'strictEqual'
    }
    
    Node.js v19.0.0-pre
  ...


Reason report/test-report-fatalerror-oomerror-not-set
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44809/, https://github.com/nodejs/node/pull/44741/)
Appeared test-joyent-smartos20-x64-4, test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/46912/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46931/
Example
not ok 147 report/test-report-fatalerror-oomerror-not-set
  ---
  duration_ms: 300.117
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Git Failure
Progress
  • Backing channel 'JNLP4-connect connection from ... is disconnected. (5)
  • Failed to trigger node-test-commit (8)
  • error: remote GitHub already exists. (4)
  • Error: [WinError 5] Access is denied: 'C:\\Users\\Administrator\\clcache\\stats.txt.new' -> 'C:\\Users\\Administrator\\clcache\\stats.txt' (3)
  • ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? (2)
  • pummel/test-fs-largefile (6)
  • parallel/test-worker-heap-snapshot (5)
  • sequential/test-http-regr-gh-2928 (3)
  • sequential/test-watch-mode-inspect (3)
  • sequential/test-worker-prof (3)
  • abort/test-addon-uv-handle-leak (2)
  • addons/request-interrupt/test (2)
  • parallel/test-tls-hello-parser-failure (2)
  • report/test-report-fatalerror-oomerror-not-set (2)
Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/reliability

All issues in nodejs/reliability

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.