CI Reliability 2023-03-13

Open
#522 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
git, node.js, shell

Research direction

Start with the generated ncu-ci walk pr --stats=true --markdown report and its linked Jenkins output. Investigate the failures at test/cctest/test_inspector_socket.cc:323 and parallel/test-http-remove-connection-header-persists-connection, alongside the listed workspace, Git, and build errors. Done means the affected CI failures have identified root causes and no longer recur.

Written by the indexing model from the issue text.

Description

Failures in node-test-pull-request/50246 to node-test-pull-request/50340 that failed more than 2 PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2023-03-13 00:15 0 2 32 5 61 2.11%
CCTest Failure
Reason ../test/cctest/test_inspector_socket.cc:323: Failure
Type CC_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/46941/, https://github.com/nodejs/node/pull/46790/)
Appeared test-softlayer-alpine312_container-x64-1, test-digitalocean-alpine311_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/50250/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50261/
Example
../test/cctest/test_inspector_socket.cc:323: Failure
Value of: (!expectation.read_expected)
  Actual: true
Expected: false
[  FAILED  ] InspectorSocketTest.HostIPv6NonRoutable (5000 ms)

Jenkins Failure
Reason Cannot read file /home/iojs/.config/jgit/config
Type JENKINS_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/47017/, https://github.com/nodejs/node/pull/47033/, https://github.com/nodejs/node/pull/46878/, https://github.com/nodejs/node/pull/46812/, https://github.com/nodejs/node/pull/47050/, https://github.com/nodejs/node/pull/46947/, https://github.com/nodejs/node/pull/47035/)
Appeared test-osuosl-aix72-ppc64_be-2
First CI https://ci.nodejs.org/job/node-test-pull-request/50310/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50336/
Example
java.io.IOException: Cannot read file /home/iojs/.config/jgit/config
org.eclipse.jgit.errors.ConfigInvalidException: Invalid line in config file
	at org.eclipse.jgit.lib.Config.fromTextRecurse(Config.java:1206)
	at org.eclipse.jgit.lib.Config.fromText(Config.java:1134)
	at org.eclipse.jgit.storage.file.FileBasedConfig.lambda$0(FileBasedConfig.java:138)
	at org.eclipse.jgit.util.FileUtils.readWithRetries(FileUtils.java:722)

Reason Build timed out (after 60 minutes). Marking the build as failed.
Type JENKINS_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/46615/, https://github.com/nodejs/node/pull/47017/, https://github.com/nodejs/node/pull/47028/, https://github.com/nodejs/node/pull/47039/)
Appeared test-softlayer-ubi81_container-x64-1, test-rackspace-win2012r2_vs2019-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/50289/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50340/
Example
Build timed out (after 60 minutes). Marking the build as failed.

Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/46809/, https://github.com/nodejs/node/pull/46815/, https://github.com/nodejs/node/pull/47050/)
Appeared test-digitalocean-freebsd12-x64-1, test-equinix-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/50269/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50332/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 45.55.90.237/45.55.90.237:46714' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at jdk.proxy2/jdk.proxy2.$Proxy209.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207)

Reason Cannot retrieve the node's workspace for hudson.matrix.MatrixProject@5b15074c[node-test-commit-osx]. Most probably the node is offline
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/46947/, https://github.com/nodejs/node/pull/46809/)
Appeared test-orka-macos10.14-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/50329/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50330/
Example
java.io.IOException: Cannot retrieve the node's workspace for hudson.matrix.MatrixProject@5b15074c[node-test-commit-osx]. Most probably the node is offline
	at hudson.matrix.MatrixRun$MatrixRunExecution.getParentWorkspaceLease(MatrixRun.java:173)
	at hudson.matrix.MatrixRun$MatrixRunExecution.decideWorkspace(MatrixRun.java:187)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:509)
	at hudson.model.Run.execute(Run.java:1897)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:153)

Git Failure
Build Failure
Reason FATAL: java.nio.channels.ClosedChannelException
Type BUILD_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/47017/, https://github.com/nodejs/node/pull/47030/, https://github.com/nodejs/node/pull/46809/, https://github.com/nodejs/node/pull/46692/, https://github.com/nodejs/node/pull/46561/, https://github.com/nodejs/node/pull/41439/)
Appeared test-equinix-ubuntu2204-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/50292/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50297/
Example
FATAL: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 145.40.96.123/145.40.96.123:43896
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
		at hudson.remoting.Request.call(Request.java:199)
		at hudson.remoting.Channel.call(Channel.java:999)

Reason Failed to trigger fanned build
Type BUILD_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/47030/, https://github.com/nodejs/node/pull/46809/, https://github.com/nodejs/node/pull/46692/, https://github.com/nodejs/node/pull/46561/, https://github.com/nodejs/node/pull/41439/)
Appeared undefined
First CI https://ci.nodejs.org/job/node-test-pull-request/50293/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50297/
Example
Failed to trigger fanned build

Reason error: remote GitHub already exists.
Type BUILD_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/46979/, https://github.com/nodejs/node/pull/46815/, https://github.com/nodejs/node/pull/46947/)
Appeared test-rackspace-win2012r2_vs2019-x64-1, test-rackspace-win2012r2_vs2017-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/50248/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50300/
Example
error: remote GitHub already exists.

> git fetch --prune --no-tags GitHub 

> git fetch --prune --no-tags https://github.com/nodejs/build.git +refs/heads/main:refs/remotes/jenkins_tmp

Reason error: The last gc run reported the following. Please correct the root cause
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/47001/, https://github.com/nodejs/node/pull/46969/)
Appeared test-ibm-ubuntu1804-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/50263/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50266/
Example
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

warning: There are too many unreachable loose objects; run 'git prune' to remove them.

Reason fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/46947/, https://github.com/nodejs/node/pull/46692/)
Appeared test-equinix-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/50331/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50339/
Example
fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository
fatal: Could not read from remote repository.
fatal: failed to run repack

JSTest Failure
Reason parallel/test-http-remove-connection-header-persists-connection
Type JS_TEST_FAILURE
Failed PR 10 (https://github.com/nodejs/node/pull/46979/, https://github.com/nodejs/node/pull/47010/, https://github.com/nodejs/node/pull/46615/, https://github.com/nodejs/node/pull/47017/, https://github.com/nodejs/node/pull/47030/, https://github.com/nodejs/node/pull/46809/, https://github.com/nodejs/node/pull/46692/, https://github.com/nodejs/node/pull/46561/, https://github.com/nodejs/node/pull/41439/, https://github.com/nodejs/node/pull/46947/)
Appeared test-orka-macos11-x64-1, test-equinix-ubuntu2004_sharedlibs_container-arm64-5, test-equinix-ubuntu2004_sharedlibs_container-arm64-6
First CI https://ci.nodejs.org/job/node-test-pull-request/50248/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50329/
Example
not ok 1194 parallel/test-http-remove-connection-header-persists-connection
  ---
  duration_ms: 0.238
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:125
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    false !== true
    
        at /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/parallel/test-http-remove-connection-header-persists-connection.js:67:16
        at Timeout._onTimeout (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/parallel/test-http-remove-connection-header-persists-connection.js:54:7)
        at listOnTimeout (node:internal/timers:573:17)
        at process.processTimers (node:internal/timers:514:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: 'strictEqual'
    }
    
    Node.js v20.0.0-pre
  ...


Reason parallel/test-worker-nearheaplimit-deadlock
Type JS_TEST_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/46815/, https://github.com/nodejs/node/pull/47017/, https://github.com/nodejs/node/pull/47030/, https://github.com/nodejs/node/pull/46809/, https://github.com/nodejs/node/pull/46692/, https://github.com/nodejs/node/pull/46561/, https://github.com/nodejs/node/pull/41439/)
Appeared test-softlayer-ubuntu1804_sharedlibs_container-x64-3, test-ibm-rhel8-x64-1, test-digitalocean-ubuntu1804_sharedlibs_container-x64-8
First CI https://ci.nodejs.org/job/node-test-pull-request/50290/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50297/
Example
not ok 3516 parallel/test-worker-nearheaplimit-deadlock
  ---
  duration_ms: 120.33
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason sequential/test-child-process-pass-fd
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/46979/, https://github.com/nodejs/node/pull/46658/, https://github.com/nodejs/node/pull/46692/)
Appeared test-macstadium-macos11.0-arm64-3, test-macstadium-macos11.0-arm64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/50248/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50301/
Example
not ok 3780 sequential/test-child-process-pass-fd
  ---
  duration_ms: 0.254
  severity: crashed
  exitcode: -10
  stack: |-
  ...


Reason sequential/test-http-regr-gh-2928
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/46969/, https://github.com/nodejs/node/pull/46812/, https://github.com/nodejs/node/pull/47039/)
Appeared test-digitalocean-freebsd12-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/50267/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50338/
Example
not ok 3863 sequential/test-http-regr-gh-2928
  ---
  duration_ms: 120.47
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    .+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.
  ...


Reason parallel/test-fs-watch-recursive-add-file-with-url
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/46812/, https://github.com/nodejs/node/pull/46947/)
Appeared test-ibm-rhel8-s390x-1, test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/50299/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50300/
Example
not ok 1272 parallel/test-fs-watch-recursive-add-file-with-url
  ---
  duration_ms: 120.121
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-fs-watch-recursive-add-folder
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/47012/, https://github.com/nodejs/node/pull/46947/)
Appeared test-ibm-rhel8-s390x-1, test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/50278/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50300/
Example
not ok 1271 parallel/test-fs-watch-recursive-add-folder
  ---
  duration_ms: 120.17
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-http-remove-header-stays-removed
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/46812/, https://github.com/nodejs/node/pull/46947/)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-5, test-digitalocean-ubuntu1804_sharedlibs_container-x64-1, test-digitalocean-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-alpine311_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/50320/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50326/
Example
not ok 1372 parallel/test-http-remove-header-stays-removed
  ---
  duration_ms: 0.232
  severity: fail
  exitcode: 1
  stack: |-
    ok
    node:assert:125
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    false !== true
    
        at Timeout._onTimeout (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-http-remove-header-stays-removed.js:62:14)
        at listOnTimeout (node:internal/timers:573:17)
        at process.processTimers (node:internal/timers:514:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: 'strictEqual'
    }
    
    Node.js v20.0.0-pre
  ...


Reason parallel/test-tls-ticket-cluster
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/46692/, https://github.com/nodejs/node/pull/47033/)
Appeared test-rackspace-win2012r2_vs2015-x64-2, test-ibm-rhel8-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/50295/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50311/
Example
not ok 749 parallel/test-tls-ticket-cluster
  ---
  duration_ms: 0.864
  severity: fail
  exitcode: 1
  stack: |-
    [primary] got "listening"
    [primary] connecting 65207 session? false
    [primary] got "listening"
    [primary] got "listening"
    [primary] got "listening"
    [worker] connection reused? false
    [primary] got "not-reused"
    [primary] connecting 65207 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 65207 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 65207 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 65207 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 65207 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 65207 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connect...

Reason sequential/test-debugger-object-type-remote-object
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/46979/, https://github.com/nodejs/node/pull/46812/)
Appeared test-macstadium-macos11.0-arm64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/50248/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50320/
Example
not ok 3818 sequential/test-debugger-object-type-remote-object
  ---
  duration_ms: 3.877
  severity: fail
  exitcode: 1
  stack: |-
    Mismatched noop function calls. Expected exactly 1, actual 0.
        at Proxy.mustCall (/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/test/common/index.js:407:10)
        at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/test/sequential/test-debugger-object-type-remote-object.js:42:14)
        at Module._compile (node:internal/modules/cjs/loader:1287:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1341:10)
        at Module.load (node:internal/modules/cjs/loader:1145:32)
        at Module._load (node:internal/modules/cjs/loader:984:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
        at node:internal/main/run_main_module:23:47
  ...


Reason sequential/test-debugger-pid
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/46979/, https://github.com/nodejs/node/pull/46812/)
Appeared test-macstadium-macos11.0-arm64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/50248/
Last CI https://ci.nodejs.org/job/node-test-pull-request/50320/
Example
not ok 3819 sequential/test-debugger-pid
  ---
  duration_ms: 5.631
  severity: fail
  exitcode: 1
  stack: |-
    node:internal/process/promises:289
                triggerUncaughtException(err, true /* fromPromise */);
                ^
    
    AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Timeout (5000) while waiting for /break/; found: Warning: script 'alive.js' was not loaded yet.
    
    debug> 
    Break on start in test/fixtures/debugger/three-lines.js:1
    > 1 let x = 1;
      2 x = x + 1;
      3 module.exports = x;
    debug> 
        at runTest (/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/test/sequential/test-debugger-pid.js:28:12)
        at Timeout.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/test/common/debugger.js:90:18)
        at listOnTimeout (node:internal/timers:573:17)
        at process.processTimers (node:internal/timers:514:7) {
      generatedMessage: false,
      code: 'ERR_ASSERTION',
      actual: Error: Ti...

Progress
  • ../test/cctest/test_inspector_socket.cc:323: Failure (2)
  • Cannot read file /home/iojs/.config/jgit/config (7)
  • Build timed out (after 60 minutes). Marking the build as failed. (4)
  • Backing channel 'JNLP4-connect connection from ... is disconnected. (3)
  • Cannot retrieve the node's workspace for hudson.matrix.MatrixProject@5b15074c[node-test-commit-osx]. Most probably the node is offline (2)
  • FATAL: java.nio.channels.ClosedChannelException (6)
  • Failed to trigger fanned build (5)
  • error: remote GitHub already exists. (3)
  • error: The last gc run reported the following. Please correct the root cause (2)
  • fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository (2)
  • parallel/test-http-remove-connection-header-persists-connection (10)
  • parallel/test-worker-nearheaplimit-deadlock (7)
  • sequential/test-child-process-pass-fd (3)
  • sequential/test-http-regr-gh-2928 (3)
  • parallel/test-fs-watch-recursive-add-file-with-url (2)
  • parallel/test-fs-watch-recursive-add-folder (2)
  • parallel/test-http-remove-header-stays-removed (2)
  • parallel/test-tls-ticket-cluster (2)
  • sequential/test-debugger-object-type-remote-object (2)
  • sequential/test-debugger-pid (2)
Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/reliability

All issues in nodejs/reliability

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.