nodejs / nodejs/reliability

CI Reliability 2024-03-28

Open
#819 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Description

Failures in node-test-pull-request/57878 to node-test-pull-request/57977 that failed 2 or more PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2024-03-28 00:08 1 3 19 2 75 3.09%
Jenkins Failure
Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/51362/, https://github.com/nodejs/node/pull/51306/)
Appeared test-digitalocean-freebsd12-x64-2, test-orka-macos10.15-x64-2, test-softlayer-debian12-x64-1, test-rackspace-fedora32-x64-1, test-digitalocean-fedora32-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/57939/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57953/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 107.170.28.213/107.170.28.213:31156' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at jdk.proxy2/jdk.proxy2.$Proxy141.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1212)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1204)

CCTest Failure
Git Failure
Build Failure
Reason fatal: No rebase in progress?
Type BUILD_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/52147/, https://github.com/nodejs/node/pull/52165/, https://github.com/nodejs/node/pull/51362/, https://github.com/nodejs/node/pull/51306/, https://github.com/nodejs/node/pull/52190/)
Appeared test-ibm-ubuntu2204_sharedlibs_container-x64-4, test-digitalocean-freebsd12-x64-2, test-digitalocean-fedora39-x64-1, test-ibm-ubuntu2204_sharedlibs_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/57897/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57977/
Example
fatal: No rebase in progress?

Reason Failed to trigger sub builds
Type BUILD_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/51362/, https://github.com/nodejs/node/pull/52153/, https://github.com/nodejs/node/pull/52135/)
Appeared undefined
First CI https://ci.nodejs.org/job/node-test-pull-request/57923/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57973/
Example
Failed to trigger sub builds

Reason fatal error: /Library/Developer/CommandLineTools/usr/bin/libtool: can't write to output file (No space left on device)
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52227/, https://github.com/nodejs/node/pull/51783/)
Appeared test-macstadium-macos11.0-arm64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/57970/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57971/
Example
fatal error: /Library/Developer/CommandLineTools/usr/bin/libtool: can't write to output file (No space left on device)

JSTest Failure
Reason parallel/test-http2-large-write-multiple-requests
Type JS_TEST_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/52143/, https://github.com/nodejs/node/pull/52199/, https://github.com/nodejs/node/pull/51362/, https://github.com/nodejs/node/pull/52217/, https://github.com/nodejs/node/pull/52215/, https://github.com/nodejs/node/pull/51821/)
Appeared test-ibm-rhel8-x64-1, test-orka-macos11-x64-2, test-ibm-ubuntu2204_sharedlibs_container-x64-2, test-digitalocean-rhel8-x64-1, test-digitalocean-alpine318_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/57912/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57967/
Example
not ok 2167 parallel/test-http2-large-write-multiple-requests
  ---
  duration_ms: 120077.05900
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    server sends content 1
    server sends content 2
    server sends content 3
    server sends content 4
    server sends content 5
    server sends content 6
    server sends content 7
    server sends content 8
    server sends content 9
    server sends content 10
    server sends content 11
    server sends content 12
    server sends content 13
    server sends content 14
    server sends content 15
    server sends content 16
    server sends content 17
    server sends content 18
    server sends content 19
    server sends content 20
    server sends content 21
    server sends content 22
    server sends content 23
    server sends content 24
    server sends content 25
    server sends content 26
    server sends content 27
    server sends content 28
    server sends content 29
    server sends content 30
    server sends content 31
    server s...

Reason parallel/test-performance-function
Type JS_TEST_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/52184/, https://github.com/nodejs/node/pull/52164/, https://github.com/nodejs/node/pull/52212/, https://github.com/nodejs/node/pull/51127/, https://github.com/nodejs/node/pull/52190/)
Appeared undefined, test-ibm-ubuntu2204-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/57885/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57962/
Example
not ok 2404 parallel/test-performance-function
  ---
  duration_ms: 329.90200
  severity: fail
  exitcode: 1
  stack: |-
    
    node:internal/event_target:1096
      process.nextTick(() => { throw err; });
                               ^
    RangeError [ERR_OUT_OF_RANGE]: The value of "val" is out of range. It must be >= 1 && <= 9007199254740991. Received 0
        at RecordableHistogram.record (node:internal/histogram:291:5)
        at processComplete (node:internal/perf/timerify:40:15)
        at timerified m (node:internal/perf/timerify:87:5)
        at /home/iojs/build/workspace/node-test-commit-custom-suites-freestyle/test/parallel/test-performance-function.js:103:5 {
      code: 'ERR_OUT_OF_RANGE'
    }
    
    Node.js v22.0.0-pre
  ...


Reason parallel/test-http-get-pipeline-problem
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/52165/, https://github.com/nodejs/node/pull/51362/, https://github.com/nodejs/node/pull/52164/, https://github.com/nodejs/node/pull/52175/)
Appeared test-azure_msft-win2016_vs2017-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/57927/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57958/
Example
not ok 356 parallel/test-http-get-pipeline-problem
  ---
  duration_ms: 354.00500
  severity: fail
  exitcode: 3221226505
  stack: |-
    image.length = 45658
    recv 0
    done 0
    recv 1
    done 1
    recv 2
    done 2
    recv 3
    done 3
    recv 4
    done 4
    recv 5
    recv 6
    done 5
    done 6
    recv 7
    done 7
    recv 8
  ...


Reason sequential/test-worker-fshandles-error-on-termination
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/52183/, https://github.com/nodejs/node/pull/52165/, https://github.com/nodejs/node/pull/51362/, https://github.com/nodejs/node/pull/52164/)
Appeared test-rackspace-win2019_vs2019-x64-1, test-rackspace-win2019_vs2019-x64-3, test-azure_msft-win10_vs2019-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/57886/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57940/
Example
not ok 936 sequential/test-worker-fshandles-error-on-termination
  ---
  duration_ms: 665.03200
  severity: crashed
  exitcode: 3221225477
  stack: |-
  ...


Reason benchmark/test-benchmark-http2
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/52184/, https://github.com/nodejs/node/pull/52164/, https://github.com/nodejs/node/pull/52165/)
Appeared test-azure_msft-win2016_vs2017-x64-1, test-azure_msft-win2016_vs2017-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/57901/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57920/
Example
not ok 184 benchmark/test-benchmark-http2
  ---
  duration_ms: 2070.00500
  severity: fail
  exitcode: 1
  stack: |-
    Error: test-double-http2 failed with 3221226505.
        at ChildProcess.<anonymous> (C:\workspace\node-test-binary-windows-native-suites\node\benchmark\_http-benchmarkers.js:242:16)
        at Object.onceWrapper (node:events:632:26)
        at ChildProcess.emit (node:events:517:28)
        at maybeClose (node:internal/child_process:1098:16)
        at ChildProcess._handle.onexit (node:internal/child_process:303:5)
    node:assert:125
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    3221226505 !== 0
    
        at ChildProcess.<anonymous> (C:\workspace\node-test-binary-windows-native-suites\node\test\common\benchmark.js:28:12)
        at ChildProcess.emit (node:events:517:28)
        at ChildProcess._handle.onexit (node:internal/child_process:292:12) {
      generatedMessage: true,
      code: 'ERR_ASSERTIO...

Reason sequential/test-http-regr-gh-2928
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/52184/, https://github.com/nodejs/node/pull/52143/, https://github.com/nodejs/node/pull/52165/)
Appeared test-rackspace-win2016_vs2015-x64-2, test-digitalocean-freebsd12-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/57887/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57913/
Example
not ok 963 sequential/test-http-regr-gh-2928
  ---
  duration_ms: 382.99700
  severity: fail
  exitcode: 3221226505
  stack: |-
    .
  ...


Reason wpt/test-url
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/52147/, https://github.com/nodejs/node/pull/52183/, https://github.com/nodejs/node/pull/52175/)
Appeared test-digitalocean-ubuntu2204_sharedlibs_container-x64-7, test-ibm-ubuntu2204_sharedlibs_container-x64-5, test-ibm-ubuntu2204_sharedlibs_container-x64-4, test-ibm-ubuntu2204_sharedlibs_container-x64-3, test-azure_msft-win11_vs2022-x64-3, test-digitalocean-ubuntu2204_sharedlibs_container-x64-5
First CI https://ci.nodejs.org/job/node-test-pull-request/57883/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57958/
Example
not ok 4056 wpt/test-url
  ---
  duration_ms: 3135.10300
  severity: fail
  exitcode: 1
  stack: |-
    [SKIPPED] idlharness-shadowrealm.window.js: ShadowRealm support is not enabled
    [SKIPPED] javascript-urls.window.js: requires document.body reference
    [SKIPPED] percent-encoding.window.js: TODO: port from .window.js
    [SKIPPED] url-setters-a-area.window.js?include=file: already tested in url-setters.any.js
    [SKIPPED] url-setters-a-area.window.js?include=javascript: already tested in url-setters.any.js
    [SKIPPED] url-setters-a-area.window.js?include=mailto: already tested in url-setters.any.js
    [SKIPPED] url-setters-a-area.window.js?exclude=(file|javascript|mailto): already tested in url-setters.any.js
    [EXPECTED_FAILURE][FAIL] searchParams on location object
    assert_false: location object should not have a searchParams attribute expected false got true
        at Test.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/fixtures/wpt/url/historical.any.js:3:5...

Reason parallel/test-blob-file-backed
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52153/, https://github.com/nodejs/node/pull/52143/)
Appeared test-digitalocean-fedora39-x64-1, test-ibm-ubuntu2204_sharedlibs_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/57893/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57912/
Example
not ok 330 parallel/test-blob-file-backed
  ---
  duration_ms: 137.45500
  severity: fail
  exitcode: 1
  stack: |-
    
    node:internal/blob:291
              const error = lazyDOMException('The blob could not be read', 'NotReadableError');
                            ^
    DOMException [NotReadableError]: The blob could not be read
        at BlobReader.<anonymous> (node:internal/blob:291:25)
    
    Node.js v22.0.0-pre
  ...


Reason parallel/test-fs-read-stream-concurrent-reads
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52143/, https://github.com/nodejs/node/pull/51362/)
Appeared test-ibm-ubi81_container-x64-1, test-digitalocean-ubuntu2204_sharedlibs_container-x64-1, test-rackspace-win2022_vs2022-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/57912/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57931/
Example
not ok 1702 parallel/test-fs-read-stream-concurrent-reads
  ---
  duration_ms: 120066.30700
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-fs-watch-recursive-update-file
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52182/, https://github.com/nodejs/node/pull/51820/)
Appeared test-equinix-debian11_container-armv7l-1, test-equinix-rhel8_container-arm64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/57917/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57976/
Example
not ok 988 parallel/test-fs-watch-recursive-update-file
  ---
  duration_ms: 534.08400
  severity: fail
  exitcode: 1
  stack: |-
    Mismatched <anonymous> function calls. Expected at least 1, actual 0.
        at Proxy.mustCallAtLeast (/home/iojs/build/workspace/node-test-binary-armv7l/test/common/index.js:443:10)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-binary-armv7l/test/parallel/test-fs-watch-recursive-update-file.js:33:29)
        at Module._compile (node:internal/modules/cjs/loader:1421:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1499:10)
        at Module.load (node:internal/modules/cjs/loader:1232:32)
        at Module._load (node:internal/modules/cjs/loader:1048:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:187:14)
        at node:internal/main/run_main_module:28:49
  ...


Reason parallel/test-macos-app-sandbox
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52217/, https://github.com/nodejs/node/pull/52190/)
Appeared test-macstadium-macos11.0-arm64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/57959/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57962/
Example
not ok 1899 parallel/test-macos-app-sandbox
  ---
  duration_ms: 1170.55400
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:126
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    1 !== 0
    
        at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/test/parallel/test-macos-app-sandbox.js:47:8)
        at Module._compile (node:internal/modules/cjs/loader:1421:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1499:10)
        at Module.load (node:internal/modules/cjs/loader:1232:32)
        at Module._load (node:internal/modules/cjs/loader:1048:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:187:14)
        at node:internal/main/run_main_module:28:49 {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 1,
      expected: 0,
      operator: 'strictEqual'
    }
    
    Node.js v22.0.0-pre
  ...


Reason parallel/test-tls-ticket-cluster
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52165/, https://github.com/nodejs/node/pull/52143/)
Appeared test-azure_msft-win10_vs2019-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/57910/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57912/
Example
not ok 762 parallel/test-tls-ticket-cluster
  ---
  duration_ms: 566.99700
  severity: fail
  exitcode: 1
  stack: |-
    [primary] got "listening"
    [primary] connecting 51915 session? false
    [primary] got "listening"
    [primary] got "listening"
    [primary] connecting 51915 session? false
    [worker] connection reused? false
    [primary] got "not-reused"
    [primary] connecting 51915 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 51915 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 51915 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 51915 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 51915 session? true
    [primary] got "listening"
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 51915 session? true
    [worker] connection reused? tru...

Reason pummel/test-crypto-timing-safe-equal-benchmarks
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/51362/, https://github.com/nodejs/node/pull/52190/)
Appeared test-azure_msft-win11_vs2022-x64-4, test-macstadium-macos11.0-arm64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/57926/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57962/
Example
not ok 937 pummel/test-crypto-timing-safe-equal-benchmarks
  ---
  duration_ms: 4172.97600
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:408
        throw err;
        ^
    
    AssertionError [ERR_ASSERTION]: timingSafeEqual should not leak information from its execution time (t=7.224298851491582)
        at Object.<anonymous> (c:\workspace\node-test-binary-windows-js-suites\node\test\pummel\test-crypto-timing-safe-equal-benchmarks.js:109:1)
        at Module._compile (node:internal/modules/cjs/loader:1421:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1499:10)
        at Module.load (node:internal/modules/cjs/loader:1232:32)
        at Module._load (node:internal/modules/cjs/loader:1048:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:187:14)
        at node:internal/main/run_main_module:28:49 {
      generatedMessage: false,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '=='
    }
    ...

Reason wpt/test-wasm-webapi
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52135/, https://github.com/nodejs/node/pull/52175/)
Appeared test-digitalocean-ubuntu2204_sharedlibs_container-x64-7, test-ibm-ubuntu2204-x64-1, test-ibm-ubuntu2204_sharedlibs_container-x64-5, test-ibm-ubuntu2204_sharedlibs_container-x64-4, test-ibm-ubuntu2204_sharedlibs_container-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/57947/
Last CI https://ci.nodejs.org/job/node-test-pull-request/57958/
Example
not ok 4058 wpt/test-wasm-webapi
  ---
  duration_ms: 823.03700
  severity: fail
  exitcode: 1
  stack: |-
    [SKIPPED] abort.any.js: WPTRunner does not support fetch()
    [SKIPPED] contenttype.any.js: WPTRunner does not support fetch()
    [SKIPPED] historical.any.js: indexedDB is not defined
    [SKIPPED] idlharness.any.js: not configured
    [SKIPPED] instantiateStreaming-bad-imports.any.js: Flaky on ARM with V8 >= 11.2
    [SKIPPED] origin.sub.any.js: CORS not implemented
    [SKIPPED] status.any.js: WPTRunner does not support fetch()
    [UNEXPECTED_FAILURE][FAIL] compileStreaming: no body
    "fatal" option is not supported on Node.js compiled without ICU
    TypeError [ERR_NO_ICU]: "fatal" option is not supported on Node.js compiled without ICU
        at new TextDecoder (node:internal/encoding:487:17)
        at lib/web/websocket/util.js (node:internal/deps/undici/undici:10923:23)
        at __require (node:internal/deps/undici/undici:6:50)
        at lib/web/websocket/websocket.js (node:internal/de...

Progress
  • Backing channel 'JNLP4-connect connection from ... is disconnected. (2)
  • fatal: No rebase in progress? (5)
  • Failed to trigger sub builds (3)
  • fatal error: /Library/Developer/CommandLineTools/usr/bin/libtool: can't write to output file (No space left on device) (2)
  • parallel/test-http2-large-write-multiple-requests (6)
  • parallel/test-performance-function (5)
  • parallel/test-http-get-pipeline-problem (4)
  • sequential/test-worker-fshandles-error-on-termination (4)
  • benchmark/test-benchmark-http2 (3)
  • sequential/test-http-regr-gh-2928 (3)
  • wpt/test-url (3)
  • parallel/test-blob-file-backed (2)
  • parallel/test-fs-read-stream-concurrent-reads (2)
  • parallel/test-fs-watch-recursive-update-file (2)
  • parallel/test-macos-app-sandbox (2)
  • parallel/test-tls-ticket-cluster (2)
  • pummel/test-crypto-timing-safe-equal-benchmarks (2)
  • wpt/test-wasm-webapi (2)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the generated report and the linked Jenkins console logs, using ncu-ci walk pr --stats=true --markdown to understand how failures are collected. Review the named tests, including parallel/test-http2-large-write-multiple-requests, parallel/test-performance-function, and parallel/test-http-get-pipeline-problem. Done means the recurring CI failures have a documented resolution or no longer reproduce across CI runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, shell
Domain
ci-cd, devops, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.