CI Reliability 2023-11-24

未关闭
#728 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
25/100
Issue 类型
缺陷
描述清晰度
需要澄清
活跃度
停滞
技术栈
javascript, node.js, shell

调研方向

从链接的 CI 控制台日志开始,查看 parallel/test-worker-nearheaplimit-deadlock、parallel/test-fs-watch-recursive-add-file-to-existing-subfolder 以及列出的其他测试,并比较它们在所列平台上的失败情况。使用 ncu-ci walk pr --stats=true 报告的上下文来识别可复现的模式。当报告的 CI 失败已得到理解,并且受影响的测试在列出的环境中不再间歇性失败时,即视为完成。

由索引模型根据 Issue 内容生成。

描述

Failures in node-test-pull-request/55784 to node-test-pull-request/55882 that failed 2 or more PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2023-11-24 00:13 0 14 17 0 69 14.00%
JSTest Failure
Reason parallel/test-worker-nearheaplimit-deadlock
Type JS_TEST_FAILURE
Failed PR 17 (https://github.com/nodejs/node/pull/50789/, https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50759/, https://github.com/nodejs/node/pull/50828/, https://github.com/nodejs/node/pull/50779/, https://github.com/nodejs/node/pull/50661/, https://github.com/nodejs/node/pull/49156/, https://github.com/nodejs/node/pull/50709/, https://github.com/nodejs/node/pull/50834/, https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50853/, https://github.com/nodejs/node/pull/50794/, https://github.com/nodejs/node/pull/50858/, https://github.com/nodejs/node/pull/50758/, https://github.com/nodejs/node/pull/50846/, https://github.com/nodejs/node/pull/50860/, https://github.com/nodejs/node/pull/50825/)
Appeared test-nearform-macos11.0-arm64-1, test-macstadium-macos11.0-arm64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/55787/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55882/
Example
not ok 3262 parallel/test-worker-nearheaplimit-deadlock
  ---
  duration_ms: 173.04800
  severity: crashed
  exitcode: -6
  stack: |-
    
    <--- Last few GCs --->
    
    [49547:0x280008000]       28 ms: Mark-Compact 1.7 (6.0) -> 1.7 (6.0) MB, 1.46 / 0.00 ms  (average mu = 0.321, current mu = 0.027) background allocation failure; GC in old space requested
    [49547:0x280008000]       30 ms: Mark-Compact 1.7 (6.0) -> 1.7 (6.0) MB, 2.12 / 0.00 ms  (average mu = 0.197, current mu = 0.019) background allocation failure; GC in old space requested
    
    
    <--- JS stacktrace --->
    
    FATAL ERROR: LocalHeap: allocation failed Allocation failed - JavaScript heap out of memory
    ----- Native stack trace -----
    
     1: 0x104c97db8 node::Abort() [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/out/Release/node]
     2: 0x104c97fb8 node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx1...

Reason parallel/test-fs-watch-recursive-add-file-to-existing-subfolder
Type JS_TEST_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/50828/, https://github.com/nodejs/node/pull/50715/, https://github.com/nodejs/node/pull/50661/, https://github.com/nodejs/node/pull/49156/, https://github.com/nodejs/node/pull/49884/, https://github.com/nodejs/node/pull/50860/)
Appeared test-ibm-rhel8-s390x-1, test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/55793/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55874/
Example
not ok 1365 parallel/test-fs-watch-recursive-add-file-to-existing-subfolder
  ---
  duration_ms: 120112.84500
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-blob-file-backed
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/50791/, https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/50773/)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-9, test-digitalocean-alpine318_container-x64-1, test-digitalocean-ubi81_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55784/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55838/
Example
not ok 278 parallel/test-blob-file-backed
  ---
  duration_ms: 501.47400
  severity: fail
  exitcode: 1
  stack: |-
    
    node:internal/blob:291
              const error = lazyDOMException('The blob could not be read', 'NotReadableError');
                            ^
    DOMException [NotReadableError]: The blob could not be read
        at BlobReader.<anonymous> (node:internal/blob:291:25)
    
    Node.js v22.0.0-pre
  ...


Reason parallel/test-debugger-run-after-quit-restart
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50661/, https://github.com/nodejs/node/pull/49884/)
Appeared test-ibm-aix72-ppc64_be-1, test-ibm-aix72-ppc64_be-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55801/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55876/
Example
not ok 777 parallel/test-debugger-run-after-quit-restart
  ---
  duration_ms: 5464.20400
  severity: fail
  exitcode: 1
  stack: |-
    /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/test/common/debugger.js:90
              reject(new Error([
                     ^
    
    Error: Timeout (5000) while waiting for /break (?:on start )?in/i; found: 
        at Timeout.<anonymous> (/home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-ppc64/test/common/debugger.js:90:18)
        at listOnTimeout (node:internal/timers:573:17)
        at process.processTimers (node:internal/timers:514:7)
    
    Node.js v22.0.0-pre
  ...


Reason parallel/test-fs-watch-recursive-symlink
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50759/, https://github.com/nodejs/node/pull/50820/)
Appeared test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/55791/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55803/
Example
not ok 1369 parallel/test-fs-watch-recursive-symlink
  ---
  duration_ms: 120099.65800
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-inspector-scriptparsed-context
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50715/, https://github.com/nodejs/node/pull/50840/)
Appeared test-ibm-ubi81_container-x64-1, test-ibm-rhel8-s390x-1
First CI https://ci.nodejs.org/job/node-test-pull-request/55810/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55844/
Example
not ok 2255 parallel/test-inspector-scriptparsed-context
  ---
  duration_ms: 15338.84100
  severity: fail
  exitcode: 1
  stack: |-
    [test] Connecting to a child Node process
    [test] Testing /json/list
    [err] Debugger listening on ws://127.0.0.1:35069/5219d0d7-8f76-4233-b8bd-770bf90ff872
    [err] 
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    [err] Debugger attached.
    [err] 
    Timed out waiting for matching notification (break on [eval]:2)
    1
  ...


Reason parallel/test-inspector-stop-profile-after-done
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50834/, https://github.com/nodejs/node/pull/50858/)
Appeared test-ibm-rhel8-s390x-1, test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/55850/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55868/
Example
not ok 2419 parallel/test-inspector-stop-profile-after-done
  ---
  duration_ms: 120765.27000
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    [test] Connecting to a child Node process
    [test] Testing /json/list
    [err] Debugger listening on ws://127.0.0.1:36693/73fa285e-ca53-4b86-9759-e5498c2ce303
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    [err] Debugger attached.
    [err] 
  ...


Reason parallel/test-runner-inspect
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50828/, https://github.com/nodejs/node/pull/49884/)
Appeared test-ibm-rhel8-s390x-4, test-ibm-rhel8-s390x-1
First CI https://ci.nodejs.org/job/node-test-pull-request/55793/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55849/
Example
not ok 3665 parallel/test-runner-inspect
  ---
  duration_ms: 120006.77300
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    [test] Connecting to a child Node process
    [test] Testing /json/list
    [err] (node:3432802) Warning: Using the inspector with --test forces running at a concurrency of 1. Use the inspectPort option to run with concurrency
    [err] (Use `node --trace-warnings ...` to show where the warning was created)
    [err] 
    [out] TAP version 13
    [out] 
    [err] Debugger listening on ws://127.0.0.1:41245/abefdb08-aac7-4f6c-9b3a-0acdc4622ce8
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    [err] Debugger attached.
    [err] 
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    [out] # Debugger ending on ws://127.0.0.1:41245/abefdb08-aac7-4f6c-9b3a-0acdc4622ce8
    [out] 
  ...


Reason parallel/test-string-decoder
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/50820/)
Appeared test-rackspace-win2016_vs2015-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/55794/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55803/
Example
not ok 717 parallel/test-string-decoder
  ---
  duration_ms: 420.99400
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:635
          throw err;
          ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
    + actual - expected
    
    + Comparison {}
    - Comparison {
    -   code: 'ERR_STRING_TOO_LONG'
    - }
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-string-decoder.js:205:10)
        at Module._compile (node:internal/modules/cjs/loader:1376:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
        at Module.load (node:internal/modules/cjs/loader:1207:32)
        at Module._load (node:internal/modules/cjs/loader:1023:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:12)
        at node:internal/main/run_main_module:28:49 {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: RangeError: Array buffer alloc...

Jenkins Failure
Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50833/)
Appeared test-digitalocean-fedora38-x64-2, test-orka-macos11-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55813/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55834/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 134.209.172.40/134.209.172.40:41402' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at jdk.proxy2/jdk.proxy2.$Proxy166.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1212)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1204)

Reason Build timed out (after 120 minutes). Marking the build as failed.
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50651/, https://github.com/nodejs/node/pull/50758/)
Appeared test-digitalocean-fedora38-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55817/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55870/
Example
Build timed out (after 120 minutes). Marking the build as failed.

Git Failure
Build Failure
Reason fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository
Type BUILD_FAILURE
Failed PR 12 (https://github.com/nodejs/node/pull/50787/, https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50759/, https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50790/, https://github.com/nodejs/node/pull/50709/, https://github.com/nodejs/node/pull/50840/, https://github.com/nodejs/node/pull/50834/, https://github.com/nodejs/node/pull/50846/, https://github.com/nodejs/node/pull/50858/, https://github.com/nodejs/node/pull/50860/)
Appeared test-equinix-ubuntu2004_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55788/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55881/
Example
fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository
fatal: Could not read from remote repository.
fatal: failed to run repack

Reason ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Type BUILD_FAILURE
Failed PR 8 (https://github.com/nodejs/node/pull/50791/, https://github.com/nodejs/node/pull/50709/, https://github.com/nodejs/node/pull/50651/, https://github.com/nodejs/node/pull/50594/, https://github.com/nodejs/node/pull/48882/, https://github.com/nodejs/node/pull/50763/, https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50860/)
Appeared test-equinix-ubuntu2004_container-armv7l-1, test-equinix-ubuntu2004_container-armv7l-2, test-osuosl-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/55784/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55874/
Example
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
[PostBuildScript] - [INFO] Executing post build scripts.
[node-test-binary-armv7l] $ /bin/bash -ex /tmp/jenkins8683146355660303296.sh
+ '[' -d .git ']'
+ git clean -fdx
Removing out/

Reason fatal error: Killed signal terminated program cc1plus
Type BUILD_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/50820/, https://github.com/nodejs/node/pull/50831/, https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50833/, https://github.com/nodejs/node/pull/49846/, https://github.com/nodejs/node/pull/49884/)
Appeared test-digitalocean-rhel8_arm_cross_container-x64-2, test-digitalocean-fedora37-x64-1, test-digitalocean-alpine318_container-x64-1, test-digitalocean-rhel8_arm_cross_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/55803/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55839/
Example
fatal error: Killed signal terminated program cc1plus

Reason error: The last gc run reported the following. Please correct the root cause
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/34111/, https://github.com/nodejs/node/pull/50730/)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-1, test-ibm-ubuntu1804-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/55798/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55878/
Example
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

warning: There are too many unreachable loose objects; run 'git prune' to remove them.

Reason error: remote GitHub already exists.
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/50354/, https://github.com/nodejs/node/pull/50653/)
Appeared test-rackspace-win2022_vs2022-x64-1, test-rackspace-win2022_vs2022-x64-3, test-rackspace-win2022_vs2022-x64-2, test-rackspace-win2022_vs2022-x64-5
First CI https://ci.nodejs.org/job/node-test-pull-request/55851/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55857/
Example
error: remote GitHub already exists.

> git fetch --prune --no-tags GitHub 

> git fetch --prune --no-tags https://github.com/nodejs/build.git +refs/heads/main:refs/remotes/jenkins_tmp

CCTest Failure
undefined
Reason Unknown
Type undefined
Failed PR 11 (https://github.com/nodejs/node/pull/50828/, https://github.com/nodejs/node/pull/50825/, https://github.com/nodejs/node/pull/49156/, https://github.com/nodejs/node/pull/49793/, https://github.com/nodejs/node/pull/50730/, https://github.com/nodejs/node/pull/50651/, https://github.com/nodejs/node/pull/50833/, https://github.com/nodejs/node/pull/49846/, https://github.com/nodejs/node/pull/49884/, https://github.com/nodejs/node/pull/50858/, https://github.com/nodejs/node/pull/50758/)
Appeared test-rackspace-centos7-x64-1, test-equinix-centos7_container-arm64-1, test-equinix-centos7_container-arm64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/55793/
Last CI https://ci.nodejs.org/job/node-test-pull-request/55870/
Example
Unknown

Progress
  • parallel/test-worker-nearheaplimit-deadlock (17)
  • parallel/test-fs-watch-recursive-add-file-to-existing-subfolder (6)
  • parallel/test-blob-file-backed (3)
  • parallel/test-debugger-run-after-quit-restart (2)
  • parallel/test-fs-watch-recursive-symlink (2)
  • parallel/test-inspector-scriptparsed-context (2)
  • parallel/test-inspector-stop-profile-after-done (2)
  • parallel/test-runner-inspect (2)
  • parallel/test-string-decoder (2)
  • Backing channel 'JNLP4-connect connection from ... is disconnected. (2)
  • Build timed out (after 120 minutes). Marking the build as failed. (2)
  • fatal: '/home/iojs/.ccache/node.shared.reference' does not appear to be a git repository (12)
  • ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? (8)
  • fatal error: Killed signal terminated program cc1plus (6)
  • error: The last gc run reported the following. Please correct the root cause (2)
  • error: remote GitHub already exists. (2)
  • Unknown (11)
主要语言
Shell
星标
57
派生
20
PR 合并指标
30 天内没有已合并 PR

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

nodejs/reliability 的其他 Issue

查看 nodejs/reliability 的全部 Issue

相似的 Issue

更多 Shell/Bash Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。