CI Reliability 2024-05-01
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- nodejs, shell
- Domain
- build-system, ci-cd, testing-qa
Research direction
Start with the generated ncu-ci report and its linked Jenkins console logs, especially the repeated timeout, Windows compiler, workspace, rebase, and JavaScript test failures. Compare the affected CI jobs and failure messages to determine which recurring reliability problem is actionable. Done is not defined by the issue beyond addressing the reported CI failures.
Written by the indexing model from the issue text.
Description
Failures in node-test-pull-request/58727 to node-test-pull-request/58825 that failed 2 or more PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)
| UTC Time | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
|---|---|---|---|---|---|---|
| 2024-05-01 00:09 | 0 | 5 | 22 | 5 | 68 | 5.26% |
Jenkins Failure
| Reason | Build timed out (after 60 minutes). Marking the build as failed. |
|---|---|
| Type | JENKINS_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/52714/, https://github.com/nodejs/node/pull/52751/) |
| Appeared | test-ibm-ubi81_container-x64-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58800/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58809/ |
Example
Build timed out (after 60 minutes). Marking the build as failed.
Git Failure
Build Failure
| Reason | error C1060: compiler is out of heap space [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_initializers.vcxproj] |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 8 (https://github.com/nodejs/node/pull/52701/, https://github.com/nodejs/node/pull/52719/, https://github.com/nodejs/node/pull/52713/, https://github.com/nodejs/node/pull/52723/, https://github.com/nodejs/node/pull/52674/, https://github.com/nodejs/node/pull/52727/, https://github.com/nodejs/node/pull/52627/, https://github.com/nodejs/node/pull/52646/) |
| Appeared | test-rackspace-win2022_vs2022-x64-6, test-rackspace-win2022_vs2022-x64-1, test-rackspace-win2022_vs2022-x64-3, test-rackspace-win2022_vs2022-x64-2, test-rackspace-win2022_vs2022-x64-4 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58729/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58808/ |
Example
error C1060: compiler is out of heap space [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_initializers.vcxproj]
heap-object-tq-csa.cc
js-atomics-synchronization-tq-csa.cc
js-array-buffer-tq-csa.cc
js-array-tq-csa.cc
| Reason | ERROR: No files found with the specified search criteria. |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 5 (https://github.com/nodejs/node/pull/52719/, https://github.com/nodejs/node/pull/52714/, https://github.com/nodejs/node/pull/52751/, https://github.com/nodejs/node/pull/52545/, https://github.com/nodejs/node/pull/52706/) |
| Appeared | test-rackspace-win2022_vs2022-x64-6, test-rackspace-win2022_vs2022-x64-5, test-rackspace-win2022_vs2022-x64-1, test-rackspace-win2022_vs2022-x64-3 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58754/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58817/ |
Example
ERROR: No files found with the specified search criteria.
1 file(s) copied.
> set DEBUG_HELPER=1
> call vcbuild.bat arm64 release msi
| Reason | ERROR: Step ‘Record fingerprints of files to track usage’ failed: no workspace for ... |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 3 (https://github.com/nodejs/node/pull/52691/, https://github.com/nodejs/node/pull/52713/, https://github.com/nodejs/node/pull/52215/) |
| Appeared | test-digitalocean-ubuntu2204-x64-2, test-rackspace-fedora32-x64-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58740/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58749/ |
Example
ERROR: Step ‘Record fingerprints of files to track usage’ failed: no workspace for node-test-commit-linux/nodes=ubuntu2204-64 #57507
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE
| Reason | fatal: No rebase in progress? |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 3 (https://github.com/nodejs/node/pull/52047/, https://github.com/nodejs/node/pull/52751/, https://github.com/nodejs/node/pull/52545/) |
| Appeared | test-ibm-ubuntu2204_sharedlibs_container-x64-2, test-digitalocean-ubuntu2204_sharedlibs_container-x64-10, test-ibm-ubi81_container-x64-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58739/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58807/ |
Example
fatal: No rebase in progress?
| Reason | Failed to trigger sub builds |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/51575/, https://github.com/nodejs/node/pull/52583/) |
| Appeared | undefined |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58783/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58811/ |
Example
Failed to trigger sub builds
| Reason | fatal error: Killed signal terminated program cc1plus |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/52657/, https://github.com/nodejs/node/pull/52713/) |
| Appeared | undefined, test-digitalocean-ubuntu2204-x64-2 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58747/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58762/ |
Example
fatal error: Killed signal terminated program cc1plus
CCTest Failure
JSTest Failure
Example
not ok 942 parallel/test-error-serdes
---
duration_ms: 120217.64000
severity: fail
exitcode: 1
stack: |-
timeout
...
Example
not ok 4013 benchmark/test-benchmark-crypto
---
duration_ms: 2311.16600
severity: fail
exitcode: 1
stack: |-
node:assert:126
throw new AssertionError(obj);
^
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
1 !== 0
at main (/home/iojs/build/workspace/node-test-commit-linux/benchmark/crypto/timingSafeEqual.js:21:10)
at /home/iojs/build/workspace/node-test-commit-linux/benchmark/common.js:54:9
at process.processTicksAndRejections (node:internal/process/task_queues:77:11) {
generatedMessage: true,
code: 'ERR_ASSERTION',
actual: 1,
expected: 0,
operator: 'strictEqual'
}
Node.js v23.0.0-pre
node:assert:126
throw new AssertionError(obj);
^
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
1 !== 0
at ChildProcess.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/test/common/benchmark.js:28:12)
a...
Example
not ok 3025 parallel/test-http2-large-write-multiple-requests
---
duration_ms: 120149.28000
severity: fail
exitcode: -15
stack: |-
timeout
server sends content 1
server sends content 2
server sends content 3
server sends content 4
server sends content 5
server sends content 6
server sends content 7
server sends content 8
server sends content 9
server sends content 10
server sends content 11
server sends content 12
server sends content 13
server sends content 14
server sends content 15
server sends content 16
server sends content 17
server sends content 18
server sends content 19
server sends content 20
server sends content 21
server sends content 22
server sends content 23
server sends content 24
server sends content 25
server sends content 26
server sends content 27
server sends content 28
server sends content 29
server sends content 30
server sends content 31
server s...
Example
not ok 1904 parallel/test-macos-app-sandbox
---
duration_ms: 6418.62000
severity: fail
exitcode: 1
stack: |-
node:assert:126
throw new AssertionError(obj);
^
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
1 !== 0
at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/parallel/test-macos-app-sandbox.js:47:8)
at Module._compile (node:internal/modules/cjs/loader:1480:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1564:10)
at Module.load (node:internal/modules/cjs/loader:1287:32)
at Module._load (node:internal/modules/cjs/loader:1103:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:168:12)
at node:internal/main/run_main_module:30:49 {
generatedMessage: true,
code: 'ERR_ASSERTION',
actual: 1,
expected: 0,
operator: 'strictEqual'
}
Node.js v23.0.0-pre
...
| Reason | parallel/test-net-write-fully-async-buffer |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 4 (https://github.com/nodejs/node/pull/52164/, https://github.com/nodejs/node/pull/52723/, https://github.com/nodejs/node/pull/52329/, https://github.com/nodejs/node/pull/52646/) |
| Appeared | test-equinix-rhel8_container-arm64-1, test-equinix-ubuntu2004_container-arm64-2 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58745/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58808/ |
Example
not ok 3960 parallel/test-net-write-fully-async-buffer
---
duration_ms: 360097.78200
severity: fail
exitcode: -15
stack: |-
timeout
...
| Reason | pummel/test-crypto-timing-safe-equal-benchmarks |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 4 (https://github.com/nodejs/node/pull/52714/, https://github.com/nodejs/node/pull/52719/, https://github.com/nodejs/node/pull/52723/, https://github.com/nodejs/node/pull/52674/) |
| Appeared | test-macstadium-macos11.0-arm64-4 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58733/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58772/ |
Example
not ok 3781 pummel/test-crypto-timing-safe-equal-benchmarks
---
duration_ms: 1659.01300
severity: fail
exitcode: 1
stack: |-
node:assert:408
throw err;
^
AssertionError [ERR_ASSERTION]: timingSafeEqual should not leak information from its execution time (t=-4.802924630409026)
at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/test/pummel/test-crypto-timing-safe-equal-benchmarks.js:109:1)
at Module._compile (node:internal/modules/cjs/loader:1476:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1555:10)
at Module.load (node:internal/modules/cjs/loader:1288:32)
at Module._load (node:internal/modules/cjs/loader:1104:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:191:14)
at node:internal/main/run_main_module:30:49 {
generatedMessage: false,
code: 'ERR_ASSERTION',
actual: false,
expected: true,
operator: '...
| Reason | parallel/test-module-loading-globalpaths |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 3 (https://github.com/nodejs/node/pull/52701/, https://github.com/nodejs/node/pull/52719/, https://github.com/nodejs/node/pull/52747/) |
| Appeared | test-orka-macos11-x64-1, test-macstadium-macos11.0-arm64-4, test-orka-macos11-x64-2 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58729/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58796/ |
Example
not ok 1888 parallel/test-module-loading-globalpaths
---
duration_ms: 1596.20400
severity: fail
exitcode: 1
stack: |-
node:fs:3015
binding.copyFile(
^
Error: ENOSPC: no space left on device, copyfile '/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/out/Release/node' -> '/Users/iojs/node-tmp/.tmp.1887/install/bin/node'
at Object.copyFileSync (node:fs:3015:11)
at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/parallel/test-module-loading-globalpaths.js:32:6)
at Module._compile (node:internal/modules/cjs/loader:1476:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1555:10)
at Module.load (node:internal/modules/cjs/loader:1288:32)
at Module._load (node:internal/modules/cjs/loader:1104:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:191:14)
at node:internal/main/run_main_module:30:49 {
errno: -2...
| Reason | pummel/test-fs-readfile-tostring-fail |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 3 (https://github.com/nodejs/node/pull/52719/, https://github.com/nodejs/node/pull/52723/, https://github.com/nodejs/node/pull/52714/) |
| Appeared | test-orka-macos11-x64-1, test-macstadium-macos11.0-arm64-4 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58754/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58770/ |
Example
not ok 3785 pummel/test-fs-readfile-tostring-fail
---
duration_ms: 192.72000
severity: fail
exitcode: 7
stack: |-
/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/pummel/test-fs-readfile-tostring-fail.js:77
throw err;
^
AssertionError [ERR_ASSERTION]: ifError got unwanted exception: ENOSPC: no space left on device, write
at Object.callback (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/pummel/test-fs-readfile-tostring-fail.js:35:37)
at callback (node:internal/streams/writable:757:21)
at onwriteError (node:internal/streams/writable:597:3)
at onwrite (node:internal/streams/writable:640:7)
at node:internal/fs/streams:513:5
at node:internal/fs/streams:455:14
at FSReqCallback.wrapper [as oncomplete] (node:fs:952:5) {
generatedMessage: false,
code: 'ERR_ASSERTION',
actual: [Error: ENOSPC: no space left on device, write] {
errno: -28,
code: 'ENOSPC'...
| Reason | sequential/test-single-executable-application-empty |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 3 (https://github.com/nodejs/node/pull/52727/, https://github.com/nodejs/node/pull/52728/, https://github.com/nodejs/node/pull/52730/) |
| Appeared | test-orka-macos11-x64-1, test-macstadium-macos11.0-arm64-4 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58773/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58784/ |
Example
not ok 4146 sequential/test-single-executable-application-empty
---
duration_ms: 18459.09700
severity: fail
exitcode: 1
stack: |-
Copied /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/out/Release/node to /Users/iojs/node-tmp/.tmp.4145/sea
Injected /Users/iojs/node-tmp/.tmp.4145/sea-prep.blob into /Users/iojs/node-tmp/.tmp.4145/sea
[process 34032]: --- stderr ---
error: /Library/Developer/CommandLineTools/usr/bin/codesign_allocate: can't write output file: /Users/iojs/node-tmp/.tmp.4145/sea.cstemp (No space left on device)
/Users/iojs/node-tmp/.tmp.4145/sea: the codesign_allocate helper tool cannot be found or used
[process 34032]: --- stdout ---
[process 34032]: status = 1, signal = null
/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/sequential/test-single-executable-application-empty.js:53
throw e;
^
Error: Cannot sign /Users/iojs/node-tmp/.tmp.4145/sea: Error: - process terminated with status 1,...
| Reason | parallel/test-file-write-stream4 |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/52691/, https://github.com/nodejs/node/pull/52747/) |
| Appeared | test-orka-macos11-x64-1, test-equinix-ubuntu2004_container-arm64-2 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58740/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58796/ |
Example
not ok 814 parallel/test-file-write-stream4
---
duration_ms: 1335.34900
severity: fail
exitcode: 1
stack: |-
node:events:498
throw er; // Unhandled 'error' event
^
Error: ENOSPC: no space left on device, write
Emitted 'error' event on WriteStream instance at:
at WriteStream.onerror (node:internal/streams/readable:1026:14)
at WriteStream.emit (node:events:520:28)
at emitErrorNT (node:internal/streams/destroy:170:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -28,
code: 'ENOSPC',
syscall: 'write'
}
Node.js v23.0.0-pre
...
| Reason | parallel/test-performance-eventlooputil |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/52714/, https://github.com/nodejs/node/pull/52627/) |
| Appeared | undefined |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58733/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58802/ |
Example
not ok 2430 parallel/test-performance-eventlooputil
---
duration_ms: 410.52200
severity: fail
exitcode: 1
stack: |-
node:internal/event_target:1095
process.nextTick(() => { throw err; });
^
AssertionError [ERR_ASSERTION]: 142.734356880188 < 142.73435694070434
at Timeout.runCalcTest (/home/iojs/build/workspace/node-test-commit-custom-suites-freestyle/test/parallel/test-performance-eventlooputil.js:85:10)
at Timeout.runCalcTest [as _onTimeout] (/home/iojs/build/workspace/node-test-commit-custom-suites-freestyle/test/common/index.js:469:15)
at listOnTimeout (node:internal/timers:575:11)
at process.processTimers (node:internal/timers:514:7) {
generatedMessage: false,
code: 'ERR_ASSERTION',
actual: false,
expected: true,
operator: '=='
}
Node.js v23.0.0-pre
...
| Reason | parallel/test-performance-function |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/52714/, https://github.com/nodejs/node/pull/52747/) |
| Appeared | undefined, test-ibm-rhel8-x64-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58737/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58796/ |
Example
not ok 2432 parallel/test-performance-function
---
duration_ms: 332.19600
severity: fail
exitcode: 1
stack: |-
node:internal/event_target:1095
process.nextTick(() => { throw err; });
^
RangeError [ERR_OUT_OF_RANGE]: The value of "val" is out of range. It must be >= 1 && <= 9007199254740991. Received 0
at RecordableHistogram.record (node:internal/histogram:291:5)
at processComplete (node:internal/perf/timerify:40:15)
at timerified m (node:internal/perf/timerify:87:5)
at /home/iojs/build/workspace/node-test-commit-custom-suites-freestyle/test/parallel/test-performance-function.js:103:5
at runNextTicks (node:internal/process/task_queues:60:5)
at listOnTimeout (node:internal/timers:540:9)
at process.processTimers (node:internal/timers:514:7) {
code: 'ERR_OUT_OF_RANGE'
}
Node.js v23.0.0-pre
...
| Reason | parallel/test-stream-readable-unpipe-resume |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/52329/, https://github.com/nodejs/node/pull/52728/) |
| Appeared | test-digitalocean-ubuntu2204_sharedlibs_container-x64-5, test-equinix-ubuntu2004_sharedlibs_container-arm64-6 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/58775/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/58787/ |
Example
not ok 3558 parallel/test-stream-readable-unpipe-resume
---
duration_ms: 120101.95300
severity: fail
exitcode: -15
stack: |-
timeout
...
Progress
-
Build timed out (after 60 minutes). Marking the build as failed.(2) -
error C1060: compiler is out of heap space [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_initializers.vcxproj](8) -
ERROR: No files found with the specified search criteria.(5) -
ERROR: Step ‘Record fingerprints of files to track usage’ failed: no workspace for ...(3) -
fatal: No rebase in progress?(3) -
Failed to trigger sub builds(2) -
fatal error: Killed signal terminated program cc1plus(2) -
parallel/test-error-serdes(13) -
benchmark/test-benchmark-crypto(7) -
parallel/test-http2-large-write-multiple-requests(7) -
parallel/test-macos-app-sandbox(7) -
parallel/test-net-write-fully-async-buffer(4) -
pummel/test-crypto-timing-safe-equal-benchmarks(4) -
parallel/test-module-loading-globalpaths(3) -
pummel/test-fs-readfile-tostring-fail(3) -
sequential/test-single-executable-application-empty(3) -
parallel/test-file-write-stream4(2) -
parallel/test-performance-eventlooputil(2) -
parallel/test-performance-function(2) -
parallel/test-stream-readable-unpipe-resume(2)
- Dominant language
- Shell
- Stars
- 57
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from nodejs/reliability
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
nodejs/reliability#1666 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
nodejs/reliability#1665 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
nodejs/reliability#1664 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
nodejs/reliability#1663 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
nodejs/reliability#1662 ·
All issues in nodejs/reliability
Similar issues
-
Update Vish to 1.1.5 Openpackage-update
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
oSoWoSo/vOid_Community_repOsitory#144 · 1 comment ·
-
help wanted new command
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
tldr-pages/tldr#24151 ·
-
PacReq: Rune IDE Openpackage request
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pacstall/pacstall-programs#9402 ·
-
[BUG] ci: the first prefetch-images.sh call runs before shard.txt exists, so it is always a no-op Openbug github-actions
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
update-request
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
msys2/MINGW-packages#31768 ·