CI Reliability 2022-10-29

Open
#414 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
node.js
Domain
ci-cd, testing-qa

Research direction

Start with the named test entry points, including test/parallel/test-watch-mode-files_watcher.js, test/pummel/test-fs-largefile.js, test/parallel/test-fs-watch.js, and the linked CI console examples. Reproduce the reported failures on the affected platforms and trace each failure to its cause. Done means the affected tests pass reliably and the CI failures are addressed or documented.

Written by the indexing model from the issue text.

Description

Failures in node-test-pull-request/47448 to node-test-pull-request/47544 that failed more than 2 PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2022-10-29 00:13 1 4 34 5 56 4.26%
JSTest Failure
Reason parallel/test-watch-mode-files_watcher
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/45153/, https://github.com/nodejs/node/pull/45165/, https://github.com/nodejs/node/pull/45091/, https://github.com/nodejs/node/pull/44020/)
Appeared test-orka-macos11-x64-1, test-softlayer-ubi81_container-x64-1, test-orka-macos11-x64-2, test-equinix-debian10_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/47459/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47523/
Example
not ok 3410 parallel/test-watch-mode-files_watcher
  ---
  duration_ms: 120.20
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    TAP version 13
    # Subtest: watch mode file watcher
        # Subtest: should watch changed files
        ok 1 - should watch changed files
          ---
          duration_ms: 133.80578
          ...


Reason pummel/test-fs-largefile
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/44935/, https://github.com/nodejs/node/pull/45068/, https://github.com/nodejs/node/pull/45098/, https://github.com/nodejs/node/pull/44412/)
Appeared test-rackspace-win2012r2_vs2019-x64-1, test-nearform-macos10.15-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47448/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47538/
Example
not ok 849 pummel/test-fs-largefile
  ---
  duration_ms: 0.939
  severity: fail
  exitcode: 1
  stack: |-
    Can't clean tmpdir: C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.850
    Files blocking: [ 'large.txt' ]
    
    node:internal/fs/utils:347
        throw err;
        ^
    
    Error: ENOSPC: no space left on device, ftruncate
        at Object.ftruncateSync (node:fs:1131:3)
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\pummel\test-fs-largefile.js:37:4)
        at Module._compile (node:internal/modules/cjs/loader:1175:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1229:10)
        at Module.load (node:internal/modules/cjs/loader:1038:32)
        at Module._load (node:internal/modules/cjs/loader:879:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:82:12)
        at node:internal/main/run_main_module:23:47 {
      errno: -4055,
      syscall: 'ftruncate',
      code: 'ENOSPC'...

Reason parallel/test-heapsnapshot-near-heap-limit-by-api-in-worker
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/45153/, https://github.com/nodejs/node/pull/45157/, https://github.com/nodejs/node/pull/44490/)
Appeared test-digitalocean-rhel8-x64-1, test-osuosl-rhel8-ppc64_le-1, test-equinix-ubuntu2004_container-arm64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47459/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47510/
Example
not ok 1193 parallel/test-heapsnapshot-near-heap-limit-by-api-in-worker
  ---
  duration_ms: 1738.25600
  severity: fail
  exitcode: 1
  stack: |-
    
    Invoked NearHeapLimitCallback, processing=false, current_limit=52428800, initial_limit=52428800
    max_young_gen_size=46399488, young_gen_size=0, old_gen_size=51154312, total_size=51154312
    Estimated available memory=9223372036746788864, estimated overhead=46399488
    Start generating Heap.20221027.194837.383723.1.001.heapsnapshot...
    Invoked NearHeapLimitCallback, processing=true, current_limit=52428800, initial_limit=52428800
    max_young_gen_size=46399488, young_gen_size=0, old_gen_size=51154312, total_size=51154312
    Estimated available memory=9223372036746731520, estimated overhead=46399488
    Not generating snapshots in nested callback. new_limit=98828288
    1/1 snapshots taken.
    Removing the near heap limit callbackWrote snapshot to /home/iojs/node-tmp/.tmp.1193/Heap.20221027.194837.383723.1.001.heapsnapshot
    
    <--- Last few GC...

Reason addons/request-interrupt/test
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45127/, https://github.com/nodejs/node/pull/45175/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/47475/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47498/
Example
not ok 3506 addons/request-interrupt/test
  ---
  duration_ms: 300.177
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason node-api/test_fatal/test2
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44412/, https://github.com/nodejs/node/pull/45175/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/47482/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47498/
Example
not ok 3607 node-api/test_fatal/test2
  ---
  duration_ms: 300.177
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-corepack-yarn-install
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45212/, https://github.com/nodejs/node/pull/45148/)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/47508/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47509/
Example
not ok 591 parallel/test-corepack-yarn-install
  ---
  duration_ms: 2.314
  severity: fail
  exitcode: 1
  stack: |-
    (node:15744) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
    (Use `node --trace-deprecation ...` to show where the warning was created)
    Segmentation fault (core dumped)
    assert.js:118
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: yarn install got error code 139
        at handleExit (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-corepack-yarn-install.js:60:10)
        at handleExit (/home/iojs/build/workspace/node-test-commit-linux-containered/test/common/index.js:380:15)
        at ChildProcess.exithandler (child_process.js:390:5)
        at ChildProcess.emit (events.js:400:28)
        at maybeClose (internal/child_process.js:1088:16)
        at Socket.<anonymous> (internal/chi...

Reason parallel/test-fs-watch
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45098/, https://github.com/nodejs/node/pull/45212/)
Appeared test-softlayer-ubuntu1804_sharedlibs_container-x64-3, test-equinix-ubuntu2004_container-armv7l-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47485/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47508/
Example
not ok 1096 parallel/test-fs-watch
  ---
  duration_ms: 0.253
  severity: fail
  exitcode: 1
  stack: |-
    node:internal/fs/watchers:252
        throw error;
        ^
    
    Error: ENOSPC: System limit for number of file watchers reached, watch '/home/iojs/node-tmp/.tmp.1096/watch2'
        at FSWatcher.<computed> (node:internal/fs/watchers:244:19)
        at Object.watch (node:fs:2303:34)
        at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-fs-watch.js:53:22)
        at Module._compile (node:internal/modules/cjs/loader:1175:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1229:10)
        at Module.load (node:internal/modules/cjs/loader:1038:32)
        at Module._load (node:internal/modules/cjs/loader:879:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:82:12)
        at node:internal/main/run_main_module:23:47 {
      errno: -28,
      syscall: 'watch',
      code: 'ENOSPC',
    ...

Reason parallel/test-inspector-esm
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45147/, https://github.com/nodejs/node/pull/44432/)
Appeared test-ibm-rhel8-s390x-3, test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/47512/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47542/
Example
not ok 1874 parallel/test-inspector-esm
  ---
  duration_ms: 15.533
  severity: fail
  exitcode: 1
  stack: |-
    [test] Connecting to a child Node process
    [test] Testing /json/list
    [err] Debugger listening on ws://127.0.0.1:41763/9b84dfd1-97d0-4d19-b371-d45c24741e60
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    [test] Verifying debugger stops on start (--inspect-brk option)
    [err] Debugger attached.
    [err] 
    Timed out waiting for matching notification (break on file:///home/iojs/build/workspace/node-test-commit-linuxone/test/fixtures/es-modules/message.mjs:0)
    1
  ...


Reason parallel/test-net-connect-memleak
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/42384/, https://github.com/nodejs/node/pull/44412/)
Appeared test-azure_msft-win2016_vs2017-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/47524/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47538/
Example
not ok 439 parallel/test-net-connect-memleak
  ---
  duration_ms: 1.109
  severity: fail
  exitcode: 1
  stack: |-
    Mismatched noop function calls. Expected exactly 1, actual 0.
        at Proxy.mustCall (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:338:10)
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-net-connect-memleak.js:33:40)
        at Module._compile (internal/modules/cjs/loader.js:1085:14)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
        at Module.load (internal/modules/cjs/loader.js:950:32)
        at Function.Module._load (internal/modules/cjs/loader.js:790:12)
        at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)
        at internal/main/run_main_module.js:17:47
  ...


Reason report/test-report-fatalerror-oomerror-set
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45098/, https://github.com/nodejs/node/pull/45146/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/47485/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47539/
Example
not ok 665 report/test-report-fatalerror-oomerror-set
  ---
  duration_ms: 300.70
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Jenkins Failure
Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/45157/, https://github.com/nodejs/node/pull/45195/, https://github.com/nodejs/node/pull/44490/)
Appeared test-digitalocean-freebsd12-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47491/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47513/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 107.170.28.213/107.170.28.213:33668' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at jdk.proxy2/jdk.proxy2.$Proxy87.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207)

Reason Build timed out (after 60 minutes). Marking the build as failed.
Type JENKINS_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/45098/, https://github.com/nodejs/node/pull/45175/, https://github.com/nodejs/node/pull/44412/)
Appeared test-softlayer-ubi81_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/47485/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47538/
Example
Build timed out (after 60 minutes). Marking the build as failed.

Reason Build timed out (after 15 minutes). Marking the build as failed.
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45175/, https://github.com/nodejs/node/pull/45222/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/47496/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47521/
Example
Build timed out (after 15 minutes). Marking the build as failed.

Reason Failed to create a temp file on C:\workspace\node-compile-windows
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/44020/, https://github.com/nodejs/node/pull/42856/)
Appeared test-rackspace-win2012r2_vs2019-x64-5, test-rackspace-win2012r2_vs2019-x64-6
First CI https://ci.nodejs.org/job/node-test-pull-request/47523/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47526/
Example
java.io.IOException: Failed to create a temp file on C:\workspace\node-compile-windows
	at hudson.FilePath.createTextTempFile(FilePath.java:1626)
	at hudson.tasks.CommandInterpreter.createScriptFile(CommandInterpreter.java:202)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:120)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
	at org.jenkins_ci.plugins.flexible_publish.builder.FailAtEndBuilder.perform(FailAtEndBuilder.java:106)

Git Failure
Build Failure
Reason error: remote GitHub already exists.
Type BUILD_FAILURE
Failed PR 8 (https://github.com/nodejs/node/pull/45153/, https://github.com/nodejs/node/pull/45117/, https://github.com/nodejs/node/pull/44490/, https://github.com/nodejs/node/pull/45209/, https://github.com/nodejs/node/pull/45148/, https://github.com/nodejs/node/pull/45221/, https://github.com/nodejs/node/pull/43525/, https://github.com/nodejs/node/pull/45222/)
Appeared test-rackspace-win2012r2_vs2019-x64-4, test-rackspace-win2012r2_vs2019-x64-5, test-rackspace-win2012r2_vs2019-x64-1, test-rackspace-win2012r2_vs2019-x64-6, test-rackspace-win2012r2_vs2017-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/47459/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47544/
Example
error: remote GitHub already exists.

> git fetch --prune --no-tags GitHub 

> git fetch --prune --no-tags https://github.com/nodejs/build.git +refs/heads/main:refs/remotes/jenkins_tmp

Progress
  • parallel/test-watch-mode-files_watcher (4)
  • pummel/test-fs-largefile (4)
  • parallel/test-heapsnapshot-near-heap-limit-by-api-in-worker (3)
  • addons/request-interrupt/test (2)
  • node-api/test_fatal/test2 (2)
  • parallel/test-corepack-yarn-install (2)
  • parallel/test-fs-watch (2)
  • parallel/test-inspector-esm (2)
  • parallel/test-net-connect-memleak (2)
  • report/test-report-fatalerror-oomerror-set (2)
  • Backing channel 'JNLP4-connect connection from ... is disconnected. (3)
  • Build timed out (after 60 minutes). Marking the build as failed. (3)
  • Build timed out (after 15 minutes). Marking the build as failed. (2)
  • Failed to create a temp file on C:\workspace\node-compile-windows (2)
  • error: remote GitHub already exists. (8)
Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/reliability

All issues in nodejs/reliability

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.