CI Reliability 2023-11-22

Open
#726 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
javascript, node.js, shell

Research direction

Start with the linked CI logs and the listed tests, including test/parallel/test-worker-nearheaplimit-deadlock.js and test/parallel/test-fs-watch-recursive-add-file-to-existing-subfolder.js. Reproduce the failures on the affected platforms and determine which failures remain; done means the reported CI failures no longer recur.

Written by the indexing model from the issue text.

Description

Failures in node-test-pull-request/55744 to node-test-pull-request/55835 that failed 2 or more PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2023-11-22 00:14 8 13 16 2 61 14.44%
JSTest Failure
Reason parallel/test-worker-nearheaplimit-deadlock
Type JS_TEST_FAILURE
Failed PR 17 (https://github.com/nodejs/node/pull/50812/, https://github.com/nodejs/node/pull/50813/, https://github.com/nodejs/node/pull/50805/, https://github.com/nodejs/node/pull/50803/, https://github.com/nodejs/node/pull/50815/, https://github.com/nodejs/node/pull/50743/, https://github.com/nodejs/node/pull/50763/, https://github.com/nodejs/node/pull/50795/, https://github.com/nodejs/node/pull/50789/, https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50759/, https://github.com/nodejs/node/pull/50828/, https://github.com/nodejs/node/pull/50779/, https://github.com/nodejs/node/pull/50661/, https://github.com/nodejs/node/pull/49156/, https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50709/)
Appeared test-nearform-macos11.0-arm64-1, test-macstadium-macos11.0-arm64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/55753/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55825/
Example
not ok 3260 parallel/test-worker-nearheaplimit-deadlock
  ---
  duration_ms: 148.32800
  severity: crashed
  exitcode: -6
  stack: |-
    
    <--- Last few GCs --->
    
    [85348:0x150008000]       24 ms: Mark-Compact 1.7 (6.0) -> 1.7 (6.0) MB, 1.92 / 0.00 ms  (average mu = 0.336, current mu = 0.000) background allocation failure; GC in old space requested
    [85348:0x150008000]       26 ms: Mark-Compact 1.7 (6.0) -> 1.7 (6.0) MB, 2.17 / 0.00 ms  (average mu = 0.198, current mu = 0.019) background allocation failure; GC in old space requested
    
    
    <--- JS stacktrace --->
    
    FATAL ERROR: LocalHeap: allocation failed Allocation failed - JavaScript heap out of memory
    ----- Native stack trace -----
    
     1: 0x104b97d40 node::Abort() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
     2: 0x104b97f40 node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx1...

Reason parallel/test-fs-watch-recursive-add-file-to-existing-subfolder
Type JS_TEST_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/50607/, https://github.com/nodejs/node/pull/50828/, https://github.com/nodejs/node/pull/50715/, https://github.com/nodejs/node/pull/50661/, https://github.com/nodejs/node/pull/49156/)
Appeared test-ibm-rhel8-s390x-1, test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/55745/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55805/
Example
not ok 1276 parallel/test-fs-watch-recursive-add-file-to-existing-subfolder
  ---
  duration_ms: 120057.51600
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-fs-watch-recursive-symlink
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/49868/, https://github.com/nodejs/node/pull/50759/, https://github.com/nodejs/node/pull/50820/)
Appeared test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/55772/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55803/
Example
not ok 1369 parallel/test-fs-watch-recursive-symlink
  ---
  duration_ms: 120099.65800
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-blob-file-backed
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50791/, https://github.com/nodejs/node/pull/50825/)
Appeared test-digitalocean-alpine318_container-x64-1, test-digitalocean-ubi81_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55784/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55794/
Example
not ok 189 parallel/test-blob-file-backed
  ---
  duration_ms: 169.51700
  severity: fail
  exitcode: 1
  stack: |-
    
    node:internal/blob:291
              const error = lazyDOMException('The blob could not be read', 'NotReadableError');
                            ^
    DOMException [NotReadableError]: The blob could not be read
        at BlobReader.<anonymous> (node:internal/blob:291:25)
    
    Node.js v22.0.0-pre
  ...


Reason parallel/test-string-decoder
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/50820/)
Appeared test-rackspace-win2016_vs2015-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/55794/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55803/
Example
not ok 717 parallel/test-string-decoder
  ---
  duration_ms: 420.99400
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:635
          throw err;
          ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
    + actual - expected
    
    + Comparison {}
    - Comparison {
    -   code: 'ERR_STRING_TOO_LONG'
    - }
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-string-decoder.js:205:10)
        at Module._compile (node:internal/modules/cjs/loader:1376:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
        at Module.load (node:internal/modules/cjs/loader:1207:32)
        at Module._load (node:internal/modules/cjs/loader:1023:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:12)
        at node:internal/main/run_main_module:28:49 {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: RangeError: Array buffer alloc...

Reason sequential/test-single-executable-application
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50805/, https://github.com/nodejs/node/pull/49884/)
Appeared test-osuosl-rhel8-ppc64_le-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55752/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55771/
Example
not ok 4042 sequential/test-single-executable-application
  ---
  duration_ms: 12191.55000
  severity: fail
  exitcode: 1
  stack: |-
    Wrote single executable preparation blob to sea-prep.blob
    node:child_process:929
        throw err;
        ^
    
    Error: Command failed: /home/iojs/node-tmp/.tmp.4041/sea -a --b=c d
        at genericNodeError (node:internal/errors:956:15)
        at wrappedFn (node:internal/errors:510:14)
        at checkExecSyncError (node:child_process:890:11)
        at execFileSync (node:child_process:926:15)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/sequential/test-single-executable-application.js:55:43)
        at Module._compile (node:internal/modules/cjs/loader:1375:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1434:10)
        at Module.load (node:internal/modules/cjs/loader:1206:32)
        at Module._load (node:internal/modules/cjs/loader:1022:12)
        at Function.executeUserEn...

Reason sequential/test-single-executable-application-disable-experimental-sea-warning
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50805/, https://github.com/nodejs/node/pull/49884/)
Appeared test-osuosl-rhel8-ppc64_le-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55752/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55771/
Example
not ok 4043 sequential/test-single-executable-application-disable-experimental-sea-warning
  ---
  duration_ms: 9483.30200
  severity: fail
  exitcode: 1
  stack: |-
    Wrote single executable preparation blob to sea-prep.blob
    node:child_process:929
        throw err;
        ^
    
    Error: Command failed: /home/iojs/node-tmp/.tmp.4042/sea -a --b=c d
        at genericNodeError (node:internal/errors:956:15)
        at wrappedFn (node:internal/errors:510:14)
        at checkExecSyncError (node:child_process:890:11)
        at execFileSync (node:child_process:926:15)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/sequential/test-single-executable-application-disable-experimental-sea-warning.js:56:43)
        at Module._compile (node:internal/modules/cjs/loader:1375:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1434:10)
        at Module.load (node:internal/modules/cjs/loader:1206:32)
        at Module._load (node:inter...

Reason sequential/test-single-executable-application-empty
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50805/, https://github.com/nodejs/node/pull/49884/)
Appeared test-osuosl-rhel8-ppc64_le-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55752/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55771/
Example
not ok 4044 sequential/test-single-executable-application-empty
  ---
  duration_ms: 8855.77200
  severity: fail
  exitcode: 1
  stack: |-
    Wrote single executable preparation blob to sea-prep.blob
    node:child_process:929
        throw err;
        ^
    
    Error: Command failed: /home/iojs/node-tmp/.tmp.4043/sea
        at genericNodeError (node:internal/errors:956:15)
        at wrappedFn (node:internal/errors:510:14)
        at checkExecSyncError (node:child_process:890:11)
        at execFileSync (node:child_process:926:15)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/sequential/test-single-executable-application-empty.js:43:1)
        at Module._compile (node:internal/modules/cjs/loader:1375:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1434:10)
        at Module.load (node:internal/modules/cjs/loader:1206:32)
        at Module._load (node:internal/modules/cjs/loader:1022:12)
        at Function.executeUserEnt...

Reason sequential/test-single-executable-application-snapshot
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50805/, https://github.com/nodejs/node/pull/49884/)
Appeared test-osuosl-rhel8-ppc64_le-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55752/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55771/
Example
not ok 4045 sequential/test-single-executable-application-snapshot
  ---
  duration_ms: 10440.51200
  severity: fail
  exitcode: 1
  stack: |-
    [process 3566957]: --- stderr ---
    
    [process 3566957]: --- stdout ---
    
    [process 3566957]: status = null, signal = SIGSEGV
    /home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/common/child_process.js:86
        throw new Error(`${failures.join('\n')}`);
        ^
    
    Error: - process terminated with status null, expected 0
    - process terminated with signal SIGSEGV, expected null
        at logAndThrow (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/common/child_process.js:86:11)
        at expectSyncExit (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/common/child_process.js:91:5)
        at spawnSyncAndExitWithoutError (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/common/child_process.js:125:10)
        at Object.<anonymous> (/h...

Reason sequential/test-single-executable-application-snapshot-and-code-cache
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50805/, https://github.com/nodejs/node/pull/49884/)
Appeared test-osuosl-rhel8-ppc64_le-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55752/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55771/
Example
not ok 4046 sequential/test-single-executable-application-snapshot-and-code-cache
  ---
  duration_ms: 10884.78400
  severity: fail
  exitcode: 1
  stack: |-
    [process 3566989]: --- stderr ---
    
    [process 3566989]: --- stdout ---
    
    [process 3566989]: status = null, signal = SIGSEGV
    /home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/common/child_process.js:86
        throw new Error(`${failures.join('\n')}`);
        ^
    
    Error: - process terminated with status null, expected 0
    - process terminated with signal SIGSEGV, expected null
        at logAndThrow (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/common/child_process.js:86:11)
        at expectSyncExit (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/common/child_process.js:91:5)
        at spawnSyncAndExitWithoutError (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/common/child_process.js:125:10)
        at Object....

Reason sequential/test-single-executable-application-use-code-cache
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50805/, https://github.com/nodejs/node/pull/49884/)
Appeared test-osuosl-rhel8-ppc64_le-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55752/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55771/
Example
not ok 4047 sequential/test-single-executable-application-use-code-cache
  ---
  duration_ms: 9240.20700
  severity: fail
  exitcode: 1
  stack: |-
    Wrote single executable preparation blob to sea-prep.blob
    node:child_process:929
        throw err;
        ^
    
    Error: Command failed: /home/iojs/node-tmp/.tmp.4046/sea -a --b=c d
        at genericNodeError (node:internal/errors:956:15)
        at wrappedFn (node:internal/errors:510:14)
        at checkExecSyncError (node:child_process:890:11)
        at execFileSync (node:child_process:926:15)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-ppc64le/test/sequential/test-single-executable-application-use-code-cache.js:56:43)
        at Module._compile (node:internal/modules/cjs/loader:1375:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1434:10)
        at Module.load (node:internal/modules/cjs/loader:1206:32)
        at Module._load (node:internal/modules/cjs/loader:1022:12)
    ...

Jenkins Failure
Build Failure
Reason fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository
Type BUILD_FAILURE
Failed PR 9 (https://github.com/nodejs/node/pull/50703/, https://github.com/nodejs/node/pull/50805/, https://github.com/nodejs/node/pull/50787/, https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50759/, https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50790/, https://github.com/nodejs/node/pull/50709/)
Appeared test-equinix-ubuntu2004_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55747/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55835/
Example
fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository
fatal: Could not read from remote repository.

Reason ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Type BUILD_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50791/, https://github.com/nodejs/node/pull/50763/, https://github.com/nodejs/node/pull/50709/, https://github.com/nodejs/node/pull/50651/, https://github.com/nodejs/node/pull/50594/, https://github.com/nodejs/node/pull/48882/)
Appeared test-equinix-ubuntu2004_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-2, test-osuosl-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/55780/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55828/
Example
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
[PostBuildScript] - [INFO] Executing post build scripts.
[node-test-binary-armv7l] $ /bin/bash -ex /tmp/jenkins3138563081706197009.sh
+ '[' -d .git ']'
+ git clean -fdx

Reason fatal error: Killed signal terminated program cc1plus
Type BUILD_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/50680/, https://github.com/nodejs/node/pull/50820/, https://github.com/nodejs/node/pull/50831/, https://github.com/nodejs/node/pull/49793/)
Appeared test-digitalocean-fedora37-x64-1, test-digitalocean-rhel8_arm_cross_container-x64-2, test-digitalocean-rhel8_arm_cross_container-x64-1, test-digitalocean-fedora38-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55777/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55812/
Example
fatal error: Killed signal terminated program cc1plus

undefined
Reason Unknown
Type undefined
Failed PR 10 (https://github.com/nodejs/node/pull/50803/, https://github.com/nodejs/node/pull/50743/, https://github.com/nodejs/node/pull/50680/, https://github.com/nodejs/node/pull/50794/, https://github.com/nodejs/node/pull/50828/, https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/49156/, https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50651/)
Appeared test-rackspace-centos7-x64-1, test-equinix-centos7_container-arm64-1, test-equinix-centos7_container-arm64-2, test-equinix_mnx-smartos18-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/55764/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55817/
Example
Unknown

CCTest Failure
Reason Value of: (!expectation.read_expected)
Type CC_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50743/, https://github.com/nodejs/node/pull/50794/)
Appeared test-digitalocean-alpine315_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/55770/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55782/
Example
Value of: (!expectation.read_expected)
  Actual: true
Expected: false

[  FAILED  ] InspectorSocketTest.HostIPv6NonRoutable (4999 ms)

Progress
  • parallel/test-worker-nearheaplimit-deadlock (17)
  • parallel/test-fs-watch-recursive-add-file-to-existing-subfolder (5)
  • parallel/test-fs-watch-recursive-symlink (3)
  • parallel/test-blob-file-backed (2)
  • parallel/test-string-decoder (2)
  • sequential/test-single-executable-application (2)
  • sequential/test-single-executable-application-disable-experimental-sea-warning (2)
  • sequential/test-single-executable-application-empty (2)
  • sequential/test-single-executable-application-snapshot (2)
  • sequential/test-single-executable-application-snapshot-and-code-cache (2)
  • sequential/test-single-executable-application-use-code-cache (2)
  • fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository (9)
  • ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? (7)
  • fatal error: Killed signal terminated program cc1plus (4)
  • Unknown (10)
  • Value of: (!expectation.read_expected) (2)
Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/reliability

All issues in nodejs/reliability

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.