nodejs / nodejs/reliability

CI Reliability 2024-05-03

Open
#852 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Description

Failures in node-test-pull-request/58781 to node-test-pull-request/58876 that failed 2 or more PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2024-05-03 00:08 5 1 23 18 53 1.30%
Jenkins Failure
Reason Build timed out (after 60 minutes). Marking the build as failed.
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52714/, https://github.com/nodejs/node/pull/52751/)
Appeared test-ibm-ubi81_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58800/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58809/
Example
Build timed out (after 60 minutes). Marking the build as failed.

Git Failure
Build Failure
Reason error C1060: compiler is out of heap space [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_initializers.vcxproj]
Type BUILD_FAILURE
Failed PR 10 (https://github.com/nodejs/node/pull/52627/, https://github.com/nodejs/node/pull/52587/, https://github.com/nodejs/node/pull/52760/, https://github.com/nodejs/node/pull/52744/, https://github.com/nodejs/node/pull/52781/, https://github.com/nodejs/node/pull/52646/, https://github.com/nodejs/node/pull/52794/, https://github.com/nodejs/node/pull/52802/, https://github.com/nodejs/node/pull/52810/, https://github.com/nodejs/node/pull/52766/)
Appeared test-rackspace-win2022_vs2022-x64-5, test-rackspace-win2022_vs2022-x64-6, test-rackspace-win2022_vs2022-x64-4, test-rackspace-win2022_vs2022-x64-3, test-rackspace-win2022_vs2022-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58802/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58876/
Example
error C1060: compiler is out of heap space [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_initializers.vcxproj]
     Creating library ..\..\out\Release\mksnapshot_host.lib and object ..\..\out\Release\mksnapshot_host.exp
  setup-heap-internal.cc
C:\workspace\node-compile-windows\node\deps\v8\src\heap\setup-heap-internal.cc(681): fatal  error C1002: compiler is out of heap space in pass 2 [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_initializers.vcxproj]
  growable-fixed-array-tq-csa.cc

Reason fatal: No rebase in progress?
Type BUILD_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/52751/, https://github.com/nodejs/node/pull/52545/, https://github.com/nodejs/node/pull/52766/, https://github.com/nodejs/node/pull/52609/)
Appeared test-ibm-ubuntu2204_sharedlibs_container-x64-1, test-ibm-ubuntu2204_sharedlibs_container-x64-4, test-ibm-ubuntu2204_sharedlibs_container-x64-2, test-digitalocean-ubuntu2204_sharedlibs_container-x64-10
First CI https://ci.nodejs.org/job/node-test-pull-request/58806/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58845/
Example
fatal: No rebase in progress?

Reason ERROR: No files found with the specified search criteria.
Type BUILD_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/52751/, https://github.com/nodejs/node/pull/52545/, https://github.com/nodejs/node/pull/52706/)
Appeared test-rackspace-win2022_vs2022-x64-6, test-rackspace-win2022_vs2022-x64-5, test-rackspace-win2022_vs2022-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58806/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58817/
Example
ERROR: No files found with the specified search criteria.
        1 file(s) copied.

> set DEBUG_HELPER=1 

> call vcbuild.bat arm64 release msi

Reason ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Type BUILD_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/52781/, https://github.com/nodejs/node/pull/52793/, https://github.com/nodejs/node/pull/52794/)
Appeared test-equinix-ubuntu1804_sharedlibs_container-arm64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58866/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58869/
Example
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

Reason Failed to trigger sub builds
Type BUILD_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/51575/, https://github.com/nodejs/node/pull/52583/, https://github.com/nodejs/node/pull/50992/)
Appeared undefined
First CI https://ci.nodejs.org/job/node-test-pull-request/58783/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58873/
Example
Failed to trigger sub builds

Reason error: remote GitHub already exists.
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52781/, https://github.com/nodejs/node/pull/52646/)
Appeared test-rackspace-win2019_vs2019-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/58866/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58867/
Example
error: remote GitHub already exists.

> git fetch --prune --no-tags GitHub 

> git fetch --prune --no-tags https://github.com/nodejs/build.git +refs/heads/main:refs/remotes/jenkins_tmp

CCTest Failure
JSTest Failure
Reason parallel/test-http2-large-write-multiple-requests
Type JS_TEST_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/52747/, https://github.com/nodejs/node/pull/52767/, https://github.com/nodejs/node/pull/52609/, https://github.com/nodejs/node/pull/52763/, https://github.com/nodejs/node/pull/52781/, https://github.com/nodejs/node/pull/52760/, https://github.com/nodejs/node/pull/52810/)
Appeared test-digitalocean-ubuntu2204_sharedlibs_container-x64-5, test-azure_msft-win11_vs2022-x64-2, test-ibm-rhel8-x64-1, test-digitalocean-ubuntu2204_sharedlibs_container-x64-7, test-ibm-ubi81_container-x64-1, test-ibm-ubuntu2204_sharedlibs_container-x64-2, test-digitalocean-ubuntu2204_sharedlibs_container-x64-6, test-ibm-rhel8-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/58796/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58874/
Example
not ok 2828 parallel/test-http2-large-write-multiple-requests
  ---
  duration_ms: 120029.05500
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    server sends content 1
    server sends content 2
    server sends content 3
    server sends content 4
    server sends content 5
    server sends content 6
    server sends content 7
    server sends content 8
    server sends content 9
    server sends content 10
    server sends content 11
    server sends content 12
    server sends content 13
    server sends content 14
    server sends content 15
    server sends content 16
    server sends content 17
    server sends content 18
    server sends content 19
    server sends content 20
    server sends content 21
    server sends content 22
    server sends content 23
    server sends content 24
    server sends content 25
    server sends content 26
    server sends content 27
    server sends content 28
    server sends content 29
    server sends content 30
    server sends content 31
    server s...

Reason benchmark/test-benchmark-crypto
Type JS_TEST_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/52739/, https://github.com/nodejs/node/pull/52728/, https://github.com/nodejs/node/pull/52347/, https://github.com/nodejs/node/pull/52747/, https://github.com/nodejs/node/pull/52646/, https://github.com/nodejs/node/pull/52763/)
Appeared test-digitalocean-ubuntu2204_sharedlibs_container-x64-10, test-digitalocean-fedora37-x64-1, test-equinix-rhel8_container-arm64-1, test-ibm-rhel9-x64-1, test-digitalocean-ubuntu2204_sharedlibs_container-x64-5, test-digitalocean-debian12-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58785/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58834/
Example
not ok 4013 benchmark/test-benchmark-crypto
  ---
  duration_ms: 10483.62700
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:126
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    1 !== 0
    
        at main (/home/iojs/build/workspace/node-test-commit-linux-containered/benchmark/crypto/timingSafeEqual.js:21:10)
        at /home/iojs/build/workspace/node-test-commit-linux-containered/benchmark/common.js:54:9
        at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 1,
      expected: 0,
      operator: 'strictEqual'
    }
    
    Node.js v23.0.0-pre
    node:assert:126
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    1 !== 0
    
        at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test...

Reason parallel/test-net-write-fully-async-buffer
Type JS_TEST_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/52646/, https://github.com/nodejs/node/pull/52763/, https://github.com/nodejs/node/pull/52768/, https://github.com/nodejs/node/pull/52781/, https://github.com/nodejs/node/pull/52794/, https://github.com/nodejs/node/pull/52802/)
Appeared test-digitalocean-ubi81_container-x64-1, test-equinix-ubuntu2004_container-arm64-2, test-equinix-ubuntu2004_sharedlibs_container-arm64-6, test-equinix-rhel8_container-arm64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58808/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58872/
Example
not ok 2515 parallel/test-net-write-fully-async-buffer
  ---
  duration_ms: 120092.79200
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-crypto-keygen-non-standard-public-exponent
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/52347/, https://github.com/nodejs/node/pull/52744/, https://github.com/nodejs/node/pull/52766/)
Appeared test-equinix-ubuntu2004_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-2, test-equinix-debian11_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58801/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58876/
Example
not ok 3802 parallel/test-crypto-keygen-non-standard-public-exponent
  ---
  duration_ms: 360081.75900
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-error-serdes
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/52723/, https://github.com/nodejs/node/pull/52730/, https://github.com/nodejs/node/pull/52727/)
Appeared test-rackspace-win2022_vs2022-x64-3, test-equinix-ubuntu2004_container-arm64-2, test-digitalocean-ubuntu2204_sharedlibs_container-x64-6, test-ibm-ubuntu2204_sharedlibs_container-x64-3, test-azure_msft-win10_vs2019-x64-1, test-rackspace-win2022_vs2022-x64-5
First CI https://ci.nodejs.org/job/node-test-pull-request/58781/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58788/
Example
not ok 942 parallel/test-error-serdes
  ---
  duration_ms: 120217.64000
  severity: fail
  exitcode: 1
  stack: |-
    timeout
  ...


Reason parallel/test-performance-function
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/52747/, https://github.com/nodejs/node/pull/52763/, https://github.com/nodejs/node/pull/52768/)
Appeared undefined
First CI https://ci.nodejs.org/job/node-test-pull-request/58796/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58832/
Example
not ok 2437 parallel/test-performance-function
  ---
  duration_ms: 409.53800
  severity: fail
  exitcode: 1
  stack: |-
    
    node:internal/event_target:1095
      process.nextTick(() => { throw err; });
                               ^
    RangeError [ERR_OUT_OF_RANGE]: The value of "val" is out of range. It must be >= 1 && <= 9007199254740991. Received 0
        at RecordableHistogram.record (node:internal/histogram:291:5)
        at processComplete (node:internal/perf/timerify:40:15)
        at timerified m (node:internal/perf/timerify:87:5)
        at /home/iojs/build/workspace/node-test-commit-custom-suites-freestyle/test/parallel/test-performance-function.js:103:5 {
      code: 'ERR_OUT_OF_RANGE'
    }
    
    Node.js v22.1.0
  ...


Reason parallel/test-runner-output
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/52609/, https://github.com/nodejs/node/pull/52213/, https://github.com/nodejs/node/pull/52793/)
Appeared test-azure_msft-win10_vs2019-x64-1, test-azure_msft-win10_vs2019-x64-3, test-azure_msft-win11_vs2022-x64-1, test-rackspace-win2016_vs2015-x64-1, test-rackspace-win2019_vs2019-x64-3, test-rackspace-win2019_vs2019-x64-2, test-rackspace-win2022_vs2022-x64-3, test-rackspace-win2022_vs2022-x64-2, test-azure_msft-win11_vs2022-arm64-2, test-azure_msft-win11_vs2022-arm64-1, test-azure_msft-win2016_vs2017-x64-2, test-rackspace-win2022_vs2022-x64-6, test-azure_msft-win11_vs2022-arm64-5, test-osuosl-aix72-ppc64_be-3
First CI https://ci.nodejs.org/job/node-test-pull-request/58810/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58868/
Example
not ok 671 parallel/test-runner-output
  ---
  duration_ms: 5884.72800
  severity: fail
  exitcode: 1
  stack: |-
    TAP version 13
    # Subtest: test runner output
        # Subtest: test-runner/output/abort.js
        ok 1 - test-runner/output/abort.js
          ---
          duration_ms: 2062.3982
          ...


Reason parallel/test-cli-node-options
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52760/, https://github.com/nodejs/node/pull/52793/)
Appeared test-rackspace-win2022_vs2022-x64-4, test-azure_msft-win2016_vs2017-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58853/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58868/
Example
not ok 122 parallel/test-cli-node-options
  ---
  duration_ms: 1573.02300
  severity: crashed
  exitcode: 2147483651
  stack: |-
    Can't clean tmpdir: C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.124
    Files blocking: [ 'node_trace.1.log' ]
    
    node:internal/child_process:421
        throw new ErrnoException(err, 'spawn');
        ^
    
    Error: spawn UNKNOWN
        at ChildProcess.spawn (node:internal/child_process:421:11)
        at spawn (node:child_process:761:9)
        at execFile (node:child_process:351:17)
        at expect (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-cli-node-options.js:119:3)
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-cli-node-options.js:76:1)
        at Module._compile (node:internal/modules/cjs/loader:1358:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
        at Module.load (node:internal/modules/cjs/loader:1208:32)
        at Mo...

Reason parallel/test-file-write-stream4
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52747/, https://github.com/nodejs/node/pull/52763/)
Appeared test-equinix-rhel8_container-arm64-1, test-orka-macos11-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58796/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58838/
Example
not ok 3875 parallel/test-file-write-stream4
  ---
  duration_ms: 360075.81700
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-macos-app-sandbox
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52730/, https://github.com/nodejs/node/pull/52047/)
Appeared test-orka-macos11-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/58784/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58792/
Example
not ok 1904 parallel/test-macos-app-sandbox
  ---
  duration_ms: 6418.62000
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:126
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    1 !== 0
    
        at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/parallel/test-macos-app-sandbox.js:47:8)
        at Module._compile (node:internal/modules/cjs/loader:1480:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1564:10)
        at Module.load (node:internal/modules/cjs/loader:1287:32)
        at Module._load (node:internal/modules/cjs/loader:1103:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:168:12)
        at node:internal/main/run_main_module:30:49 {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 1,
      expected: 0,
      operator: 'strictEqual'
    }
    
    Node.js v23.0.0-pre
  ...


Reason parallel/test-performance-eventlooputil
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/52627/, https://github.com/nodejs/node/pull/52609/)
Appeared undefined
First CI https://ci.nodejs.org/job/node-test-pull-request/58802/
Last CI https://ci.nodejs.org/job/node-test-pull-request/58827/
Example
not ok 2434 parallel/test-performance-eventlooputil
  ---
  duration_ms: 341.81800
  severity: fail
  exitcode: 1
  stack: |-
    
    node:internal/event_target:1095
      process.nextTick(() => { throw err; });
                               ^
    AssertionError [ERR_ASSERTION]: 131.10266304016113 < 131.10266305859375
        at Timeout.runCalcTest (/home/iojs/build/workspace/node-test-commit-custom-suites-freestyle/test/parallel/test-performance-eventlooputil.js:85:10)
        at Timeout.runCalcTest [as _onTimeout] (/home/iojs/build/workspace/node-test-commit-custom-suites-freestyle/test/common/index.js:469:15)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      generatedMessage: false,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '=='
    }
    
    Node.js v23.0.0-pre
  ...


Progress
  • Build timed out (after 60 minutes). Marking the build as failed. (2)
  • error C1060: compiler is out of heap space [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_initializers.vcxproj] (10)
  • fatal: No rebase in progress? (4)
  • ERROR: No files found with the specified search criteria. (3)
  • ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? (3)
  • Failed to trigger sub builds (3)
  • error: remote GitHub already exists. (2)
  • parallel/test-http2-large-write-multiple-requests (7)
  • benchmark/test-benchmark-crypto (6)
  • parallel/test-net-write-fully-async-buffer (6)
  • parallel/test-crypto-keygen-non-standard-public-exponent (3)
  • parallel/test-error-serdes (3)
  • parallel/test-performance-function (3)
  • parallel/test-runner-output (3)
  • parallel/test-cli-node-options (2)
  • parallel/test-file-write-stream4 (2)
  • parallel/test-macos-app-sandbox (2)
  • parallel/test-performance-eventlooputil (2)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the linked Jenkins console logs and comparing each repeated failure's First CI and Last CI entries in the generated report. The report is produced with ncu-ci walk pr --stats=true --markdown; no source file, test, or concrete fix is named. Done is not defined beyond investigating the listed CI failures, so a maintainer would need to specify the intended follow-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, node.js, shell
Domain
ci-cd, devops, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.