CI Reliability 2023-11-24
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- javascript, node.js, shell
- Domain
- ci-cd, devops, testing-qa
Research direction
Start with the linked CI console logs for parallel/test-worker-nearheaplimit-deadlock, parallel/test-fs-watch-recursive-add-file-to-existing-subfolder, and the other listed tests, comparing failures across the named platforms. Use the ncu-ci walk pr --stats=true report context to identify reproducible patterns. Done means the reported CI failures are understood and the affected tests no longer fail intermittently on the listed environments.
Written by the indexing model from the issue text.
Description
Failures in node-test-pull-request/55784 to node-test-pull-request/55882 that failed 2 or more PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)
| UTC Time | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
|---|---|---|---|---|---|---|
| 2023-11-24 00:13 | 0 | 14 | 17 | 0 | 69 | 14.00% |
JSTest Failure
Example
not ok 3262 parallel/test-worker-nearheaplimit-deadlock
---
duration_ms: 173.04800
severity: crashed
exitcode: -6
stack: |-
<--- Last few GCs --->
[49547:0x280008000] 28 ms: Mark-Compact 1.7 (6.0) -> 1.7 (6.0) MB, 1.46 / 0.00 ms (average mu = 0.321, current mu = 0.027) background allocation failure; GC in old space requested
[49547:0x280008000] 30 ms: Mark-Compact 1.7 (6.0) -> 1.7 (6.0) MB, 2.12 / 0.00 ms (average mu = 0.197, current mu = 0.019) background allocation failure; GC in old space requested
<--- JS stacktrace --->
FATAL ERROR: LocalHeap: allocation failed Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
1: 0x104c97db8 node::Abort() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
2: 0x104c97fb8 node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx1...
| Reason | parallel/test-fs-watch-recursive-add-file-to-existing-subfolder |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 6 (https://github.com/nodejs/node/pull/50828/, https://github.com/nodejs/node/pull/50715/, https://github.com/nodejs/node/pull/50661/, https://github.com/nodejs/node/pull/49156/, https://github.com/nodejs/node/pull/49884/, https://github.com/nodejs/node/pull/50860/) |
| Appeared | test-ibm-rhel8-s390x-1, test-ibm-rhel8-s390x-4 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55793/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55874/ |
Example
not ok 1365 parallel/test-fs-watch-recursive-add-file-to-existing-subfolder
---
duration_ms: 120112.84500
severity: fail
exitcode: -15
stack: |-
timeout
...
| Reason | parallel/test-blob-file-backed |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 3 (https://github.com/nodejs/node/pull/50791/, https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/50773/) |
| Appeared | test-digitalocean-ubuntu1804_sharedlibs_container-x64-9, test-digitalocean-alpine318_container-x64-1, test-digitalocean-ubi81_container-x64-2 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55784/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55838/ |
Example
not ok 278 parallel/test-blob-file-backed
---
duration_ms: 501.47400
severity: fail
exitcode: 1
stack: |-
node:internal/blob:291
const error = lazyDOMException('The blob could not be read', 'NotReadableError');
^
DOMException [NotReadableError]: The blob could not be read
at BlobReader.<anonymous> (node:internal/blob:291:25)
Node.js v22.0.0-pre
...
| Reason | parallel/test-debugger-run-after-quit-restart |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/50661/, https://github.com/nodejs/node/pull/49884/) |
| Appeared | test-ibm-aix72-ppc64_be-1, test-ibm-aix72-ppc64_be-2 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55801/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55876/ |
Example
not ok 777 parallel/test-debugger-run-after-quit-restart
---
duration_ms: 5464.20400
severity: fail
exitcode: 1
stack: |-
/home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/test/common/debugger.js:90
reject(new Error([
^
Error: Timeout (5000) while waiting for /break (?:on start )?in/i; found:
at Timeout.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/test/common/debugger.js:90:18)
at listOnTimeout (node:internal/timers:573:17)
at process.processTimers (node:internal/timers:514:7)
Node.js v22.0.0-pre
...
| Reason | parallel/test-fs-watch-recursive-symlink |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/50759/, https://github.com/nodejs/node/pull/50820/) |
| Appeared | test-ibm-rhel8-s390x-4 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55791/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55803/ |
Example
not ok 1369 parallel/test-fs-watch-recursive-symlink
---
duration_ms: 120099.65800
severity: fail
exitcode: -15
stack: |-
timeout
...
| Reason | parallel/test-inspector-scriptparsed-context |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/50715/, https://github.com/nodejs/node/pull/50840/) |
| Appeared | test-ibm-ubi81_container-x64-1, test-ibm-rhel8-s390x-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55810/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55844/ |
Example
not ok 2255 parallel/test-inspector-scriptparsed-context
---
duration_ms: 15338.84100
severity: fail
exitcode: 1
stack: |-
[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:35069/5219d0d7-8f76-4233-b8bd-770bf90ff872
[err]
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[err] Debugger attached.
[err]
Timed out waiting for matching notification (break on [eval]:2)
1
...
| Reason | parallel/test-inspector-stop-profile-after-done |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/50834/, https://github.com/nodejs/node/pull/50858/) |
| Appeared | test-ibm-rhel8-s390x-1, test-ibm-rhel8-s390x-4 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55850/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55868/ |
Example
not ok 2419 parallel/test-inspector-stop-profile-after-done
---
duration_ms: 120765.27000
severity: fail
exitcode: -15
stack: |-
timeout
[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:36693/73fa285e-ca53-4b86-9759-e5498c2ce303
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[err] Debugger attached.
[err]
...
| Reason | parallel/test-runner-inspect |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/50828/, https://github.com/nodejs/node/pull/49884/) |
| Appeared | test-ibm-rhel8-s390x-4, test-ibm-rhel8-s390x-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55793/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55849/ |
Example
not ok 3665 parallel/test-runner-inspect
---
duration_ms: 120006.77300
severity: fail
exitcode: -15
stack: |-
timeout
[test] Connecting to a child Node process
[test] Testing /json/list
[err] (node:3432802) Warning: Using the inspector with --test forces running at a concurrency of 1. Use the inspectPort option to run with concurrency
[err] (Use `node --trace-warnings ...` to show where the warning was created)
[err]
[out] TAP version 13
[out]
[err] Debugger listening on ws://127.0.0.1:41245/abefdb08-aac7-4f6c-9b3a-0acdc4622ce8
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[err] Debugger attached.
[err]
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[out] # Debugger ending on ws://127.0.0.1:41245/abefdb08-aac7-4f6c-9b3a-0acdc4622ce8
[out]
...
| Reason | parallel/test-string-decoder |
|---|---|
| Type | JS_TEST_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/50820/) |
| Appeared | test-rackspace-win2016_vs2015-x64-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55794/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55803/ |
Example
not ok 717 parallel/test-string-decoder
---
duration_ms: 420.99400
severity: fail
exitcode: 1
stack: |-
node:assert:635
throw err;
^
AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected
+ Comparison {}
- Comparison {
- code: 'ERR_STRING_TOO_LONG'
- }
at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-string-decoder.js:205:10)
at Module._compile (node:internal/modules/cjs/loader:1376:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
at Module.load (node:internal/modules/cjs/loader:1207:32)
at Module._load (node:internal/modules/cjs/loader:1023:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:12)
at node:internal/main/run_main_module:28:49 {
generatedMessage: true,
code: 'ERR_ASSERTION',
actual: RangeError: Array buffer alloc...
Jenkins Failure
| Reason | Backing channel 'JNLP4-connect connection from ... is disconnected. |
|---|---|
| Type | JENKINS_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50833/) |
| Appeared | test-digitalocean-fedora38-x64-2, test-orka-macos11-x64-2 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55813/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55834/ |
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 134.209.172.40/134.209.172.40:41402' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
at jdk.proxy2/jdk.proxy2.$Proxy166.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1212)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1204)
| Reason | Build timed out (after 120 minutes). Marking the build as failed. |
|---|---|
| Type | JENKINS_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/50651/, https://github.com/nodejs/node/pull/50758/) |
| Appeared | test-digitalocean-fedora38-x64-2 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55817/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55870/ |
Example
Build timed out (after 120 minutes). Marking the build as failed.
Git Failure
Build Failure
| Reason | fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 12 (https://github.com/nodejs/node/pull/50787/, https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50759/, https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50790/, https://github.com/nodejs/node/pull/50709/, https://github.com/nodejs/node/pull/50840/, https://github.com/nodejs/node/pull/50834/, https://github.com/nodejs/node/pull/50846/, https://github.com/nodejs/node/pull/50858/, https://github.com/nodejs/node/pull/50860/) |
| Appeared | test-equinix-ubuntu2004_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-2 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55788/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55881/ |
Example
fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository
fatal: Could not read from remote repository.
fatal: failed to run repack
| Reason | ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 8 (https://github.com/nodejs/node/pull/50791/, https://github.com/nodejs/node/pull/50709/, https://github.com/nodejs/node/pull/50651/, https://github.com/nodejs/node/pull/50594/, https://github.com/nodejs/node/pull/48882/, https://github.com/nodejs/node/pull/50763/, https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50860/) |
| Appeared | test-equinix-ubuntu2004_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-2, test-osuosl-ubuntu2004_container-armv7l-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55784/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55874/ |
Example
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
[PostBuildScript] - [INFO] Executing post build scripts.
[node-test-binary-armv7l] $ /bin/bash -ex /tmp/jenkins8683146355660303296.sh
+ '[' -d .git ']'
+ git clean -fdx
Removing out/
| Reason | fatal error: Killed signal terminated program cc1plus |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 6 (https://github.com/nodejs/node/pull/50820/, https://github.com/nodejs/node/pull/50831/, https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50833/, https://github.com/nodejs/node/pull/49846/, https://github.com/nodejs/node/pull/49884/) |
| Appeared | test-digitalocean-rhel8_arm_cross_container-x64-2, test-digitalocean-fedora37-x64-1, test-digitalocean-alpine318_container-x64-1, test-digitalocean-rhel8_arm_cross_container-x64-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55803/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55839/ |
Example
fatal error: Killed signal terminated program cc1plus
| Reason | error: The last gc run reported the following. Please correct the root cause |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/34111/, https://github.com/nodejs/node/pull/50730/) |
| Appeared | test-digitalocean-ubuntu1804_sharedlibs_container-x64-1, test-ibm-ubuntu1804-x64-1 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55798/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55878/ |
Example
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.
warning: There are too many unreachable loose objects; run 'git prune' to remove them.
| Reason | error: remote GitHub already exists. |
|---|---|
| Type | BUILD_FAILURE |
| Failed PR | 2 (https://github.com/nodejs/node/pull/50354/, https://github.com/nodejs/node/pull/50653/) |
| Appeared | test-rackspace-win2022_vs2022-x64-1, test-rackspace-win2022_vs2022-x64-3, test-rackspace-win2022_vs2022-x64-2, test-rackspace-win2022_vs2022-x64-5 |
| First CI | https://ci.nodejs.org/job/node-test-pull-request/55851/ |
| Last CI | https://ci.nodejs.org/job/node-test-pull-request/55857/ |
Example
error: remote GitHub already exists.
> git fetch --prune --no-tags GitHub
> git fetch --prune --no-tags https://github.com/nodejs/build.git +refs/heads/main:refs/remotes/jenkins_tmp
CCTest Failure
undefined
Example
Unknown
Progress
-
parallel/test-worker-nearheaplimit-deadlock(17) -
parallel/test-fs-watch-recursive-add-file-to-existing-subfolder(6) -
parallel/test-blob-file-backed(3) -
parallel/test-debugger-run-after-quit-restart(2) -
parallel/test-fs-watch-recursive-symlink(2) -
parallel/test-inspector-scriptparsed-context(2) -
parallel/test-inspector-stop-profile-after-done(2) -
parallel/test-runner-inspect(2) -
parallel/test-string-decoder(2) -
Backing channel 'JNLP4-connect connection from ... is disconnected.(2) -
Build timed out (after 120 minutes). Marking the build as failed.(2) -
fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository(12) -
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?(8) -
fatal error: Killed signal terminated program cc1plus(6) -
error: The last gc run reported the following. Please correct the root cause(2) -
error: remote GitHub already exists.(2) -
Unknown(11)
- Dominant language
- Shell
- Stars
- 57
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from nodejs/reliability
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
nodejs/reliability#1666 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
nodejs/reliability#1665 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
nodejs/reliability#1664 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
nodejs/reliability#1663 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
nodejs/reliability#1662 ·
All issues in nodejs/reliability
Similar issues
-
Update Vish to 1.1.5 Openpackage-update
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
oSoWoSo/vOid_Community_repOsitory#144 · 1 comment ·
-
help wanted new command
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
tldr-pages/tldr#24151 ·
-
PacReq: Rune IDE Openpackage request
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
pacstall/pacstall-programs#9402 ·
-
[BUG] ci: the first prefetch-images.sh call runs before shard.txt exists, so it is always a no-op Openbug github-actions
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
update-request
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
msys2/MINGW-packages#31768 ·