nodejs / nodejs/reliability

CI Reliability 2024-06-06

Open
#887 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Description

Failures in node-test-pull-request/59566 to node-test-pull-request/59665 that failed 2 or more PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2024-06-06 00:08 0 5 28 1 66 5.05%
Jenkins Failure
Reason Build timed out (after 60 minutes). Marking the build as failed.
Type JENKINS_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/53086/, https://github.com/nodejs/node/pull/53205/, https://github.com/nodejs/node/pull/53300/)
Appeared test-ibm-ubi81_container-x64-1, test-ibm-ubuntu2204_sharedlibs_container-x64-3, test-ibm-ubuntu2204_sharedlibs_container-x64-1, test-ibm-ubuntu2204_sharedlibs_container-x64-4, test-ibm-ubuntu2204_sharedlibs_container-x64-5
First CI https://ci.nodejs.org/job/node-test-pull-request/59598/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59636/
Example
Build timed out (after 60 minutes). Marking the build as failed.

Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/53205/, https://github.com/nodejs/node/pull/53255/)
Appeared test-azure_msft-win11_vs2022-x64-3, test-ibm-ubuntu2204_sharedlibs_container-x64-2, test-ibm-rhel8_arm_cross_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/59599/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59646/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 20.106.250.216/20.106.250.216:2049' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at jdk.proxy2/jdk.proxy2.$Proxy175.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1212)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1204)

CCTest Failure
Reason C:\workspace\node-compile-windows\node\test\cctest\test_inspector_socket.cc(323): error: Value of: (!expectation.read_expected)
Type CC_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/53175/, https://github.com/nodejs/node/pull/53255/, https://github.com/nodejs/node/pull/53348/)
Appeared test-rackspace-win2019_vs2019-x64-4, test-azure_msft-win10_vs2019-x64-2, test-azure_msft-win11_vs2022-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/59578/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59661/
Example
C:\workspace\node-compile-windows\node\test\cctest\test_inspector_socket.cc(323): error: Value of: (!expectation.read_expected)
  Actual: true
Expected: false

[  FAILED  ] InspectorSocketTest.HostIPv6NonRoutable (5001 ms)

Build Failure
Reason ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Type BUILD_FAILURE
Failed PR 18 (https://github.com/nodejs/node/pull/53228/, https://github.com/nodejs/node/pull/53213/, https://github.com/nodejs/node/pull/53227/, https://github.com/nodejs/node/pull/53177/, https://github.com/nodejs/node/pull/53229/, https://github.com/nodejs/node/pull/53057/, https://github.com/nodejs/node/pull/53226/, https://github.com/nodejs/node/pull/53114/, https://github.com/nodejs/node/pull/53183/, https://github.com/nodejs/node/pull/53233/, https://github.com/nodejs/node/pull/53189/, https://github.com/nodejs/node/pull/53175/, https://github.com/nodejs/node/pull/53154/, https://github.com/nodejs/node/pull/53188/, https://github.com/nodejs/node/pull/53086/, https://github.com/nodejs/node/pull/53205/, https://github.com/nodejs/node/pull/53232/, https://github.com/nodejs/node/pull/53300/)
Appeared test-mnx-smartos21-x64-2, test-mnx-smartos23-x64-1, test-equinix_mnx-smartos20-x64-3, test-osuosl-aix72-ppc64_be-2, test-equinix_mnx-smartos20-x64-4, test-osuosl-aix72-ppc64_be-3, test-mnx-smartos21-x64-1, test-mnx-smartos23-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/59568/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59636/
Example
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

Reason fatal: No rebase in progress?
Type BUILD_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/53183/, https://github.com/nodejs/node/pull/53086/, https://github.com/nodejs/node/pull/53205/, https://github.com/nodejs/node/pull/53305/)
Appeared test-ibm-ubuntu2204_sharedlibs_container-x64-4, test-equinix-ubuntu2204-x64-1, test-digitalocean-ubuntu2204_sharedlibs_container-x64-2, test-ibm-ubuntu2204_sharedlibs_container-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/59583/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59641/
Example
fatal: No rebase in progress?

Reason ERROR: Step ‘Record fingerprints of files to track usage’ failed: no workspace for ...
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/53086/, https://github.com/nodejs/node/pull/53205/)
Appeared test-rackspace-fedora32-x64-1, test-digitalocean-ubuntu2204-x64-2, test-ibm-alpine319_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/59598/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59599/
Example
ERROR: Step ‘Record fingerprints of files to track usage’ failed: no workspace for node-test-commit-linux/nodes=fedora-last-latest-x64 #58226
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

undefined
JSTest Failure
Reason benchmark/test-benchmark-crypto
Type JS_TEST_FAILURE
Failed PR 11 (https://github.com/nodejs/node/pull/53226/, https://github.com/nodejs/node/pull/53175/, https://github.com/nodejs/node/pull/53233/, https://github.com/nodejs/node/pull/53228/, https://github.com/nodejs/node/pull/52983/, https://github.com/nodejs/node/pull/53241/, https://github.com/nodejs/node/pull/53183/, https://github.com/nodejs/node/pull/53251/, https://github.com/nodejs/node/pull/53255/, https://github.com/nodejs/node/pull/53294/, https://github.com/nodejs/node/pull/53336/)
Appeared test-ibm-rhel8-x64-1, test-digitalocean-ubuntu2204_sharedlibs_container-x64-5, test-azure_msft-win11_vs2022-arm64-2, test-digitalocean-rhel9-x64-1, test-digitalocean-ubuntu2204_sharedlibs_container-x64-4, test-digitalocean-ubuntu2204_sharedlibs_container-x64-10, test-equinix-ubuntu2004_container-armv7l-2, test-digitalocean-debian11-x64-1, test-ibm-ubuntu2204_sharedlibs_container-x64-1, test-rackspace-win2022_vs2022-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/59579/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59658/
Example
not ok 4015 benchmark/test-benchmark-crypto
  ---
  duration_ms: 2021.67200
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:126
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    1 !== 0
    
        at main (/home/iojs/build/workspace/node-test-commit-linux/benchmark/crypto/timingSafeEqual.js:21:10)
        at /home/iojs/build/workspace/node-test-commit-linux/benchmark/common.js:54:9
        at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 1,
      expected: 0,
      operator: 'strictEqual'
    }
    
    Node.js v23.0.0-pre
    node:assert:126
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    1 !== 0
    
        at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/test/common/benchmark.js:28:12)
        a...

Reason parallel/test-performance-function
Type JS_TEST_FAILURE
Failed PR 8 (https://github.com/nodejs/node/pull/53234/, https://github.com/nodejs/node/pull/53232/, https://github.com/nodejs/node/pull/53159/, https://github.com/nodejs/node/pull/53251/, https://github.com/nodejs/node/pull/53228/, https://github.com/nodejs/node/pull/53300/, https://github.com/nodejs/node/pull/53337/, https://github.com/nodejs/node/pull/53336/)
Appeared test-ibm-ubi81_container-x64-1, undefined, test-ibm-ubuntu2204_sharedlibs_container-x64-1, test-ibm-rhel8-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/59603/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59658/
Example
not ok 2417 parallel/test-performance-function
  ---
  duration_ms: 220.39400
  severity: fail
  exitcode: 1
  stack: |-
    node:internal/histogram:291
        validateInteger(val, 'val', 1);
        ^
    
    RangeError [ERR_OUT_OF_RANGE]: The value of "val" is out of range. It must be >= 1 && <= 9007199254740991. Received 0
        at RecordableHistogram.record (node:internal/histogram:291:5)
        at processComplete (node:internal/perf/timerify:40:15)
        at timerified m (node:internal/perf/timerify:87:5)
        at /home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-performance-function.js:103:5
        at runNextTicks (node:internal/process/task_queues:60:5)
        at listOnTimeout (node:internal/timers:540:9)
        at process.processTimers (node:internal/timers:514:7) {
      code: 'ERR_OUT_OF_RANGE'
    }
    
    Node.js v23.0.0-pre
  ...


Reason sequential/test-http-server-request-timeouts-mixed
Type JS_TEST_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/53229/, https://github.com/nodejs/node/pull/53188/, https://github.com/nodejs/node/pull/53241/, https://github.com/nodejs/node/pull/52725/, https://github.com/nodejs/node/pull/53226/, https://github.com/nodejs/node/pull/53331/, https://github.com/nodejs/node/pull/53352/)
Appeared test-orka-macos11-x64-1, test-orka-macos11-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/59574/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59665/
Example
not ok 4113 sequential/test-http-server-request-timeouts-mixed
  ---
  duration_ms: 3142.57400
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:408
        throw err;
        ^
    
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(request2.completed)
    
        at Timeout._onTimeout (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/sequential/test-http-server-request-timeouts-mixed.js:108:5)
        at listOnTimeout (node:internal/timers:573:17)
        at process.processTimers (node:internal/timers:514:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '=='
    }
    
    Node.js v23.0.0-pre
  ...


Reason parallel/test-buffer-tostring-rangeerror
Type JS_TEST_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/53188/, https://github.com/nodejs/node/pull/53183/, https://github.com/nodejs/node/pull/53205/, https://github.com/nodejs/node/pull/53255/, https://github.com/nodejs/node/pull/53226/, https://github.com/nodejs/node/pull/53337/)
Appeared test-ibm-ubuntu2204_sharedlibs_container-x64-3, test-digitalocean-ubuntu2204_sharedlibs_container-x64-2, test-digitalocean-ubuntu2204_sharedlibs_container-x64-3, test-ibm-rhel8-x64-2, test-digitalocean-ubuntu2204_sharedlibs_container-x64-7, test-digitalocean-ubi81_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/59567/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59655/
Example
not ok 1474 parallel/test-buffer-tostring-rangeerror
  ---
  duration_ms: 120056.71600
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    (node:1474366) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
    (Use `node --trace-deprecation ...` to show where the warning was created)
  ...


Reason parallel/test-net-write-fully-async-buffer
Type JS_TEST_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/53177/, https://github.com/nodejs/node/pull/53232/, https://github.com/nodejs/node/pull/53251/, https://github.com/nodejs/node/pull/53336/, https://github.com/nodejs/node/pull/53355/)
Appeared test-equinix-rhel8_container-arm64-1, test-equinix-ubuntu2004_container-arm64-2, test-equinix-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/59573/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59663/
Example
not ok 3963 parallel/test-net-write-fully-async-buffer
  ---
  duration_ms: 360067.87000
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason async-hooks/test-writewrap
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/53188/, https://github.com/nodejs/node/pull/53233/, https://github.com/nodejs/node/pull/53255/, https://github.com/nodejs/node/pull/53355/)
Appeared test-ibm-ubuntu2204_sharedlibs_container-x64-5, test-digitalocean-ubuntu2204_sharedlibs_container-x64-10, test-ibm-ubuntu2204_sharedlibs_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/59567/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59664/
Example
not ok 519 async-hooks/test-writewrap
  ---
  duration_ms: 120021.33600
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-http2-large-write-multiple-requests
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/53228/, https://github.com/nodejs/node/pull/53159/, https://github.com/nodejs/node/pull/53205/, https://github.com/nodejs/node/pull/53177/)
Appeared test-digitalocean-ubuntu2204_sharedlibs_container-x64-6, test-ibm-rhel8-x64-2, test-azure_msft-win10_vs2019-x64-4, test-ibm-ubi81_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/59568/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59639/
Example
not ok 2757 parallel/test-http2-large-write-multiple-requests
  ---
  duration_ms: 120025.10200
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    server sends content 1
    server sends content 2
    server sends content 3
    server sends content 4
    server sends content 5
    server sends content 6
    server sends content 7
    server sends content 8
    server sends content 9
    server sends content 10
    server sends content 11
    server sends content 12
    server sends content 13
    server sends content 14
    server sends content 15
    server sends content 16
    server sends content 17
    server sends content 18
    server sends content 19
    server sends content 20
    server sends content 21
    server sends content 22
    server sends content 23
    server sends content 24
    server sends content 25
    server sends content 26
    server sends content 27
    server sends content 28
    server sends content 29
    server sends content 30
    server sends content 31
    server s...

Reason wpt/test-compression
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/53337/, https://github.com/nodejs/node/pull/53331/, https://github.com/nodejs/node/pull/53355/)
Appeared test-ibm-ubuntu2204_sharedlibs_container-x64-5, test-digitalocean-ubuntu2204_sharedlibs_container-x64-10, test-digitalocean-ubuntu2204_sharedlibs_container-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/59655/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59664/
Example
not ok 4199 wpt/test-compression
  ---
  duration_ms: 1440130.77700
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    [SKIPPED] compression-bad-chunks.tentative.any.js: Execution "hangs", ArrayBuffer and TypedArray is not accepted and throws, instead of rejects during writer.write
    [SKIPPED] decompression-bad-chunks.tentative.any.js: Execution "hangs", ArrayBuffer and TypedArray is not accepted and throws, instead of rejects during writer.write
    [SKIPPED] decompression-buffersource.tentative.any.js: ArrayBuffer and TypedArray is not accepted and throws, instead of rejects during writer.write
    [SKIPPED] idlharness-shadowrealm.window.js: ShadowRealm support is not enabled
    [SKIPPED] third_party/pako/pako_inflate.min.js: This is not a test file.
    [PASS] "a" should cause the constructor to throw
    [PASS] no input should cause the constructor to throw
    [PASS] non-string input should cause the constructor to throw
    [PASS] the length of deflated data should be shorter than that of t...

Reason parallel/test-timers-immediate-queue
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/53233/, https://github.com/nodejs/node/pull/53232/)
Appeared test-ibm-rhel8-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/59595/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59607/
Example
not ok 3039 parallel/test-timers-immediate-queue
  ---
  duration_ms: 119.06200
  severity: fail
  exitcode: 1
  stack: |-
    hit 3320
    node:assert:126
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    3320 !== 10
    
        at process.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/test/parallel/test-timers-immediate-queue.js:56:10)
        at process.emit (node:events:532:35) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 3320,
      expected: 10,
      operator: 'strictEqual'
    }
    
    Node.js v23.0.0-pre
  ...


Reason parallel/test-worker-memory
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/53337/, https://github.com/nodejs/node/pull/53352/)
Appeared test-azure_msft-win10_vs2019-x64-2, test-azure_msft-win11_vs2022-x64-4, test-rackspace-win2022_vs2022-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/59656/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59665/
Example
not ok 895 parallel/test-worker-memory
  ---
  duration_ms: 1540.45000
  severity: crashed
  exitcode: 3221225477
  stack: |-
    run() called with n=15 (numWorkers=4)
    run() called with n=15 (numWorkers=4)
    run() called with n=15 (numWorkers=4)
    run() called with n=15 (numWorkers=4)
    run() called with n=14 (numWorkers=4)
    run() called with n=14 (numWorkers=4)
    run() called with n=14 (numWorkers=4)
    run() called with n=14 (numWorkers=4)
    run() called with n=13 (numWorkers=4)
    run() called with n=13 (numWorkers=4)
    run() called with n=13 (numWorkers=4)
    run() called with n=12 (numWorkers=4)
    run() called with n=12 (numWorkers=4)
    run() called with n=13 (numWorkers=4)
    run() called with n=11 (numWorkers=4)
    run() called with n=12 (numWorkers=4)
    run() called with n=11 (numWorkers=4)
    run() called with n=12 (numWorkers=4)
    run() called with n=10 (numWorkers=4)
    run() called with n=11 (numWorkers=4)
    run() called with n=10 (numWorkers=4)
    run() called...

Reason parallel/test-worker-stdio-from-preload-module
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/53332/, https://github.com/nodejs/node/pull/53337/)
Appeared test-rackspace-win2016_vs2015-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/59652/
Last CI https://ci.nodejs.org/job/node-test-pull-request/59656/
Example
not ok 896 parallel/test-worker-stdio-from-preload-module
  ---
  duration_ms: 231.00900
  severity: crashed
  exitcode: 3221225477
  stack: |-
  ...


Progress
  • Build timed out (after 60 minutes). Marking the build as failed. (3)
  • Backing channel 'JNLP4-connect connection from ... is disconnected. (2)
  • C:\workspace\node-compile-windows\node\test\cctest\test_inspector_socket.cc(323): error: Value of: (!expectation.read_expected) (3)
  • ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? (18)
  • fatal: No rebase in progress? (4)
  • ERROR: Step ‘Record fingerprints of files to track usage’ failed: no workspace for ... (2)
  • benchmark/test-benchmark-crypto (11)
  • parallel/test-performance-function (8)
  • sequential/test-http-server-request-timeouts-mixed (7)
  • parallel/test-buffer-tostring-rangeerror (6)
  • parallel/test-net-write-fully-async-buffer (5)
  • async-hooks/test-writewrap (4)
  • parallel/test-http2-large-write-multiple-requests (4)
  • wpt/test-compression (3)
  • parallel/test-timers-immediate-queue (2)
  • parallel/test-worker-memory (2)
  • parallel/test-worker-stdio-from-preload-module (2)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the generated ncu-ci walk pr --stats=true report and inspect the linked Jenkins console logs for each failure category. Review test/cctest/test_inspector_socket.cc at the reported assertion, then compare the timeout, agent-disconnection, JUnit-report, rebase, and workspace failures. Done means identifying and fixing the relevant CI reliability causes and confirming the affected jobs no longer reproduce them.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
ci-cd, devops, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.