CI Reliability 2022-09-29

Open
#388 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
18/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
shell

Research direction

Start by reviewing the linked Jenkins job consoles and the listed failure categories, including the node-test-pull-request runs and their example logs. No source files or tests are named; completion would require identifying an actionable cause for the reported CI failures and confirming that the affected jobs no longer fail for that reason.

Written by the indexing model from the issue text.

Description

Failures in node-test-pull-request/46814 to node-test-pull-request/46910 that failed more than 2 PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2022-09-29 00:23 2 8 20 2 68 8.33%
Jenkins Failure
Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44690/, https://github.com/nodejs/node/pull/44766/)
Appeared test-digitalocean-freebsd12-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/46843/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46885/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 107.170.28.213/107.170.28.213:60583' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at jdk.proxy2/jdk.proxy2.$Proxy92.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207)

Reason Build timed out (after 60 minutes). Marking the build as failed.
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44797/, https://github.com/nodejs/node/pull/44803/)
Appeared test-rackspace-win2012r2_vs2019-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/46863/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46899/
Example
Build timed out (after 60 minutes). Marking the build as failed.

Git Failure
Reason ERROR: Error fetching remote repo 'jenkins_tmp'
Type GIT_FAILURE
Failed PR 8 (https://github.com/nodejs/node/pull/44633/, https://github.com/nodejs/node/pull/44741/, https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44772/, https://github.com/nodejs/node/pull/44684/, https://github.com/nodejs/node/pull/44763/, https://github.com/nodejs/node/pull/44689/, https://github.com/nodejs/node/pull/44731/)
Appeared test-softlayer-rhel8_arm_cross_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46815/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46859/
Example
ERROR: Error fetching remote repo 'jenkins_tmp'
hudson.plugins.git.GitException: Failed to fetch from binary_tmp@147.75.72.255:binary_tmp.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:1003)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1244)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1308)
	at hudson.scm.SCM.checkout(SCM.java:540)

Build Failure
Reason FATAL: Connection timed out
Type BUILD_FAILURE
Failed PR 17 (https://github.com/nodejs/node/pull/44775/, https://github.com/nodejs/node/pull/44633/, https://github.com/nodejs/node/pull/44726/, https://github.com/nodejs/node/pull/44711/, https://github.com/nodejs/node/pull/44787/, https://github.com/nodejs/node/pull/44483/, https://github.com/nodejs/node/pull/44741/, https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44797/, https://github.com/nodejs/node/pull/44348/, https://github.com/nodejs/node/pull/44799/, https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44756/, https://github.com/nodejs/node/pull/44778/, https://github.com/nodejs/node/pull/44786/, https://github.com/nodejs/node/pull/44772/)
Appeared test-equinix-ubuntu2204-x64-1, test-ibm-ubuntu1804-x64-1, test-packetnet-ubuntu1804-x64-1, test-packetnet-ubuntu1804-x64-2, test-equinix-ubuntu2204-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/46814/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46834/
Example
FATAL: Connection timed out
java.net.ConnectException: Connection timed out
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:579)
	at java.base/sun.nio.ch.Net.connect(Net.java:568)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)

Reason error: remote GitHub already exists.
Type BUILD_FAILURE
Failed PR 9 (https://github.com/nodejs/node/pull/44633/, https://github.com/nodejs/node/pull/44787/, https://github.com/nodejs/node/pull/44797/, https://github.com/nodejs/node/pull/44348/, https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44799/, https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44741/)
Appeared test-rackspace-win2012r2_vs2019-x64-3, test-rackspace-win2012r2_vs2017-x64-1, test-rackspace-win2012r2_vs2019-x64-4, test-rackspace-win2012r2_vs2019-x64-5
First CI https://ci.nodejs.org/job/node-test-pull-request/46815/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46888/
Example
error: remote GitHub already exists.

> git fetch --prune --no-tags GitHub 

> git fetch --prune --no-tags https://github.com/nodejs/build.git +refs/heads/main:refs/remotes/jenkins_tmp

Reason Failed to trigger node-test-commit
Type BUILD_FAILURE
Failed PR 8 (https://github.com/nodejs/node/pull/44772/, https://github.com/nodejs/node/pull/44746/, https://github.com/nodejs/node/pull/44809/, https://github.com/nodejs/node/pull/44697/, https://github.com/nodejs/node/pull/44693/, https://github.com/nodejs/node/pull/44690/, https://github.com/nodejs/node/pull/44689/, https://github.com/nodejs/node/pull/44686/)
Appeared test-equinix-ubuntu2204-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/46873/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46904/
Example
Failed to trigger node-test-commit

Reason Error: [WinError 5] Access is denied: 'C:\Users\Administrator\clcache\stats.txt.new' -> 'C:\Users\Administrator\clcache\stats.txt'
Type BUILD_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44677/, https://github.com/nodejs/node/pull/44692/)
Appeared test-rackspace-win2012r2_vs2019-x64-2, test-rackspace-win2012r2_vs2019-x64-5
First CI https://ci.nodejs.org/job/node-test-pull-request/46886/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46910/
Example
Error: [WinError 5] Access is denied: 'C:\\Users\\Administrator\\clcache\\stats.txt.new' -> 'C:\\Users\\Administrator\\clcache\\stats.txt'
  [1228] Failed to execute script clcache_main
  frames-tq.cc
  finalization-registry-tq.cc
  frame-arguments-tq.cc
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(523,5): error MSB6006: "clcache_main.exe" exited with code -1. [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_base_without_compiler.vcxproj]

undefined
Reason Unknown
Type undefined
Failed PR 5 (https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44348/, https://github.com/nodejs/node/pull/44799/, https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44376/)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-7, test-digitalocean-alpine311_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46820/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46830/
Example
Unknown

JSTest Failure
Reason sequential/test-watch-mode-inspect
Type JS_TEST_FAILURE
Failed PR 9 (https://github.com/nodejs/node/pull/44787/, https://github.com/nodejs/node/pull/44741/, https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44693/, https://github.com/nodejs/node/pull/44686/, https://github.com/nodejs/node/pull/44771/, https://github.com/nodejs/node/pull/44711/)
Appeared test-nearform-macos10.15-x64-2, test-nearform-macos10.15-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46819/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46857/
Example
not ok 3789 sequential/test-watch-mode-inspect
  ---
  duration_ms: 120.67
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    TAP version 13
    (node:37761) ExperimentalWarning: The test runner is an experimental feature. This feature could change at any time
    (Use `node --trace-warnings ...` to show where the warning was created)
  ...


Reason parallel/test-watch-mode
Type JS_TEST_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/44797/, https://github.com/nodejs/node/pull/44348/, https://github.com/nodejs/node/pull/44799/, https://github.com/nodejs/node/pull/44795/, https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44756/)
Appeared test-osuosl-rhel8-ppc64_le-3, test-nearform-macos11.0-arm64-1, test-nearform-macos10.15-x64-2, test-orka-macos11-x64-1, test-ibm-rhel8-s390x-4, test-osuosl-aix72-ppc64_be-4, test-equinix-ubuntu1804_sharedlibs_container-arm64-3, test-joyent-smartos20-x64-4, undefined, test-digitalocean-freebsd12-x64-2, test-digitalocean-ubi81_container-x64-1, test-softlayer-ubuntu1804_sharedlibs_container-x64-5, test-softlayer-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-1, test-digitalocean-ubuntu1804_sharedlibs_container-x64-6, test-digitalocean-ubuntu1804_sharedlibs_container-x64-4, test-digitalocean-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-8, test-osuosl-rhel8-ppc64_le-2, test-equinix-debian10_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46825/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46831/
Example
not ok 3059 parallel/test-watch-mode
  ---
  duration_ms: 5.147
  severity: fail
  exitcode: 1
  stack: |-
    TAP version 13
    # Subtest: watch mode
        # Subtest: should watch changes to a file - event loop ended
        ok 1 - should watch changes to a file - event loop ended
          ---
          duration_ms: 286.931977
          ...


Reason pummel/test-heapdump-env
Type JS_TEST_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/44797/, https://github.com/nodejs/node/pull/44348/, https://github.com/nodejs/node/pull/44795/, https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44756/, https://github.com/nodejs/node/pull/44799/)
Appeared test-nearform-macos10.15-x64-2, test-orka-macos11-x64-2, test-softlayer-alpine311_container-x64-1, test-digitalocean-alpine312_container-x64-2, test-softlayer-debian10-x64-1, test-rackspace-fedora32-x64-1, test-digitalocean-fedora34-x64-2, test-ibm-rhel8-x64-1, test-digitalocean-ubuntu1804-x64-1, test-digitalocean-freebsd12-x64-2, test-osuosl-rhel8-ppc64_le-2, test-equinix-rhel8_container-arm64-1, test-equinix-ubuntu1804_container-arm64-2, test-equinix-ubuntu2004_container-arm64-2, test-equinix-ubuntu2004_container-armv7l-1, test-softlayer-ubi81_container-x64-1, test-softlayer-ubuntu1804_sharedlibs_container-x64-5, test-digitalocean-ubuntu1804_sharedlibs_container-x64-6, test-softlayer-ubuntu1804_sharedlibs_container-x64-4, test-softlayer-ubuntu1804_sharedlibs_container-x64-1, test-digitalocean-ubuntu1804_sharedlibs_container-x64-4, test-digitalocean-ubuntu1804_sharedlibs_container-x64-8, test-osuosl-aix72-ppc64_be-3, test-joyent-smartos20-x64-3, test-ibm-rhel8-s390x-4, test-nearform-macos11.0-arm64-1, test-equinix-ubuntu1804_sharedlibs_container-arm64-3, test-equinix-debian10_container-armv7l-1, test-osuosl-rhel8-ppc64_le-3, test-orka-macos11-x64-1, test-osuosl-aix72-ppc64_be-4, test-joyent-smartos20-x64-4, test-digitalocean-ubi81_container-x64-1, test-softlayer-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46825/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46835/
Example
not ok 3380 pummel/test-heapdump-env
  ---
  duration_ms: 1.404
  severity: fail
  exitcode: 1
  stack: |-
    /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/test/common/heap.js:139
                throw new Error(
                ^
    
    Error: expected to find child { node_name: 'Node / Realm', edge_name: 'principal_realm' } in [
      <ref *1> {
        type: 'synthetic',
        name: 'Node / Environment',
        id: 4139614240,
        self_size: 2784,
        edge_count: 50,
        trace_node_id: 0,
        detachedness: 0,
        incomingEdges: [
          {
            type: 'element',
            to: [Circular *1],
            from: {
              type: 'synthetic',
              name: '',
              id: 1,
              self_size: 0,
              edge_count: 23,
              trace_node_id: 0,
              detachedness: 0,
              incomingEdges: [],
              outgoingEdges: [Array]
            },
            name: null
          }
        ],
        outgoingEdge...

Reason sequential/test-watch-mode
Type JS_TEST_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/44797/, https://github.com/nodejs/node/pull/44348/, https://github.com/nodejs/node/pull/44799/, https://github.com/nodejs/node/pull/44795/, https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44756/)
Appeared test-osuosl-rhel8-ppc64_le-3, test-nearform-macos11.0-arm64-1, test-nearform-macos10.15-x64-2, test-orka-macos11-x64-1, test-ibm-rhel8-s390x-4, test-osuosl-aix72-ppc64_be-4, test-equinix-ubuntu1804_sharedlibs_container-arm64-3, test-joyent-smartos20-x64-4, undefined, test-digitalocean-freebsd12-x64-2, test-digitalocean-ubi81_container-x64-1, test-softlayer-ubuntu1804_sharedlibs_container-x64-5, test-softlayer-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-1, test-digitalocean-ubuntu1804_sharedlibs_container-x64-6, test-digitalocean-ubuntu1804_sharedlibs_container-x64-4, test-digitalocean-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-8, test-osuosl-rhel8-ppc64_le-2, test-equinix-debian10_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46825/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46831/
Example
not ok 3786 sequential/test-watch-mode
  ---
  duration_ms: 2.772
  severity: fail
  exitcode: 1
  stack: |-
    TAP version 13
    # Subtest: watch mode
        # Subtest: should watch changes to a file - event loop ended
        ok 1 - should watch changes to a file - event loop ended
          ---
          duration_ms: 419.051407
          ...


Reason pummel/test-fs-largefile
Type JS_TEST_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/44693/, https://github.com/nodejs/node/pull/44799/, https://github.com/nodejs/node/pull/44697/, https://github.com/nodejs/node/pull/44675/, https://github.com/nodejs/node/pull/44766/, https://github.com/nodejs/node/pull/44767/)
Appeared test-rackspace-win2012r2_vs2019-x64-4, test-nearform-macos10.15-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/46869/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46907/
Example
not ok 843 pummel/test-fs-largefile
  ---
  duration_ms: 0.943
  severity: fail
  exitcode: 1
  stack: |-
    Can't clean tmpdir: C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.844
    Files blocking: [ 'large.txt' ]
    
    node:internal/fs/utils:347
        throw err;
        ^
    
    Error: ENOSPC: no space left on device, ftruncate
        at Object.ftruncateSync (node:fs:1126:3)
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\pummel\test-fs-largefile.js:37:4)
        at Module._compile (node:internal/modules/cjs/loader:1159:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
        at Module.load (node:internal/modules/cjs/loader:1037:32)
        at Module._load (node:internal/modules/cjs/loader:878:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:86:12)
        at node:internal/main/run_main_module:23:47 {
      errno: -4055,
      syscall: 'ftruncate',
      code: 'ENOSPC'...

Reason parallel/test-worker-heap-snapshot
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44796/, https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44695/)
Appeared test-rackspace-win2012r2_vs2013-x64-1, test-rackspace-win2012r2_vs2019-x64-6, test-rackspace-win2012r2_vs2017-x64-3, test-osuosl-rhel8-ppc64_le-2
First CI https://ci.nodejs.org/job/node-test-pull-request/46841/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46908/
Example
not ok 807 parallel/test-worker-heap-snapshot
  ---
  duration_ms: 1.362
  severity: fail
  exitcode: 134
  stack: |-
    FATAL ERROR: HandleScope::HandleScope Entering the V8 API without proper locking in place
  ...


Reason abort/test-addon-uv-handle-leak
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44766/, https://github.com/nodejs/node/pull/44677/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/46885/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46898/
Example
not ok 3630 abort/test-addon-uv-handle-leak
  ---
  duration_ms: 300.129
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-fs-promises-file-handle-readFile
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44731/, https://github.com/nodejs/node/pull/44498/)
Appeared test-rackspace-win2012r2_vs2019-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/46859/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46860/
Example
not ok 251 parallel/test-fs-promises-file-handle-readFile
  ---
  duration_ms: 0.313
  severity: fail
  exitcode: 1
  stack: |-
    node:internal/process/promises:288
                triggerUncaughtException(err, true /* fromPromise */);
                ^
    
    [Error: ENOSPC: no space left on device, ftruncate] {
      errno: -4055,
      code: 'ENOSPC',
      syscall: 'ftruncate'
    }
    
    Node.js v19.0.0-pre
  ...


Progress
  • Backing channel 'JNLP4-connect connection from ... is disconnected. (3)
  • Build timed out (after 60 minutes). Marking the build as failed. (2)
  • ERROR: Error fetching remote repo 'jenkins_tmp' (8)
  • FATAL: Connection timed out (17)
  • error: remote GitHub already exists. (9)
  • Failed to trigger node-test-commit (8)
  • Error: [WinError 5] Access is denied: 'C:\\Users\\Administrator\\clcache\\stats.txt.new' -> 'C:\\Users\\Administrator\\clcache\\stats.txt' (3)
  • Unknown (5)
  • sequential/test-watch-mode-inspect (9)
  • parallel/test-watch-mode (7)
  • pummel/test-heapdump-env (7)
  • sequential/test-watch-mode (7)
  • pummel/test-fs-largefile (6)
  • parallel/test-worker-heap-snapshot (4)
  • abort/test-addon-uv-handle-leak (2)
  • parallel/test-fs-promises-file-handle-readFile (2)
Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/reliability

All issues in nodejs/reliability

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.