CI Reliability 2022-10-24

Open
#409 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
node.js, shell
Domain
ci-cd, testing-qa

Research direction

Start with the generated ncu-ci walk output and the linked CI runs, then inspect the named tests: pummel/test-fs-readfile-tostring-fail, pummel/test-fs-largefile, parallel/test-macos-app-sandbox, and the listed SmartOS failures. Compare the failures across the referenced workers and determine the changes needed for these recurring CI failures to stop appearing.

Written by the indexing model from the issue text.

Description

Failures in node-test-pull-request/47333 to node-test-pull-request/47429 that failed more than 2 PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2022-10-24 00:22 1 12 13 2 72 12.37%
JSTest Failure
Reason pummel/test-fs-readfile-tostring-fail
Type JS_TEST_FAILURE
Failed PR 17 (https://github.com/nodejs/node/pull/45046/, https://github.com/nodejs/node/pull/41276/, https://github.com/nodejs/node/pull/43716/, https://github.com/nodejs/node/pull/45068/, https://github.com/nodejs/node/pull/45026/, https://github.com/nodejs/node/pull/45061/, https://github.com/nodejs/node/pull/45060/, https://github.com/nodejs/node/pull/45091/, https://github.com/nodejs/node/pull/45127/, https://github.com/nodejs/node/pull/45043/, https://github.com/nodejs/node/pull/43525/, https://github.com/nodejs/node/pull/45129/, https://github.com/nodejs/node/pull/45092/, https://github.com/nodejs/node/pull/44713/, https://github.com/nodejs/node/pull/45098/, https://github.com/nodejs/node/pull/45132/, https://github.com/nodejs/node/pull/45135/)
Appeared test-orka-macos11-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47333/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47429/
Example
not ok 3409 pummel/test-fs-readfile-tostring-fail
  ---
  duration_ms: 0.246
  severity: fail
  exitcode: 7
  stack: |-
    /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/pummel/test-fs-readfile-tostring-fail.js:73
      throw err;
      ^
    
    AssertionError [ERR_ASSERTION]: ifError got unwanted exception: ENOSPC: no space left on device, write
        at Object.callback (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/pummel/test-fs-readfile-tostring-fail.js:31:37)
        at callback (node:internal/streams/writable:555:21)
        at onwriteError (node:internal/streams/writable:418:3)
        at onwrite (node:internal/streams/writable:460:7)
        at node:internal/fs/streams:439:14
        at FSReqCallback.wrapper [as oncomplete] (node:fs:946:5) {
      generatedMessage: false,
      code: 'ERR_ASSERTION',
      actual: [Error: ENOSPC: no space left on device, write] {
        errno: -28,
        code: 'ENOSPC',
        syscall: 'write'
      },
      expe...

Reason pummel/test-fs-largefile
Type JS_TEST_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/45076/, https://github.com/nodejs/node/pull/44713/, https://github.com/nodejs/node/pull/45068/, https://github.com/nodejs/node/pull/45043/, https://github.com/nodejs/node/pull/45135/)
Appeared test-nearform-macos10.15-x64-2, test-rackspace-win2012r2_vs2019-x64-5
First CI https://ci.nodejs.org/job/node-test-pull-request/47344/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47425/
Example
not ok 3408 pummel/test-fs-largefile
  ---
  duration_ms: 0.90
  severity: fail
  exitcode: 1
  stack: |-
    node:internal/fs/utils:347
        throw err;
        ^
    
    Error: ENOSPC: no space left on device, ftruncate
        at Object.ftruncateSync (node:fs:1131:3)
        at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/test/pummel/test-fs-largefile.js:37:4)
        at Module._compile (node:internal/modules/cjs/loader:1175:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1229:10)
        at Module.load (node:internal/modules/cjs/loader:1038:32)
        at Module._load (node:internal/modules/cjs/loader:879:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:82:12)
        at node:internal/main/run_main_module:23:47 {
      errno: -28,
      syscall: 'ftruncate',
      code: 'ENOSPC'
    }
    
    Node.js v20.0.0-pre
  ...


Reason parallel/test-macos-app-sandbox
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/45061/, https://github.com/nodejs/node/pull/45060/, https://github.com/nodejs/node/pull/45043/, https://github.com/nodejs/node/pull/45091/)
Appeared test-orka-macos11-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47350/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47375/
Example
not ok 1679 parallel/test-macos-app-sandbox
  ---
  duration_ms: 5.144
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:124
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    1 !== 0
    
        at Object.<anonymous> (/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/parallel/test-macos-app-sandbox.js:43:8)
        at Module._compile (node:internal/modules/cjs/loader:1159:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
        at Module.load (node:internal/modules/cjs/loader:1037:32)
        at Module._load (node:internal/modules/cjs/loader:878:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:82:12)
        at node:internal/main/run_main_module:23:47 {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: 1,
      expected: 0,
      operator: 'strictEqual'
    }
    
    Node.js v20.0.0-pre
  ...


Reason abort/test-abort-backtrace
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45100/, https://github.com/nodejs/node/pull/43525/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/47373/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47403/
Example
not ok 3649 abort/test-abort-backtrace
  ---
  duration_ms: 300.200
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason addons/request-interrupt/test
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45076/, https://github.com/nodejs/node/pull/45098/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/47344/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47420/
Example
not ok 3504 addons/request-interrupt/test
  ---
  duration_ms: 300.168
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason node-api/test_fatal/test2
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45100/, https://github.com/nodejs/node/pull/45132/)
Appeared test-joyent-smartos20-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/47373/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47422/
Example
not ok 3603 node-api/test_fatal/test2
  ---
  duration_ms: 300.79
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-fs-promises-watch
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45068/, https://github.com/nodejs/node/pull/45098/)
Appeared test-softlayer-alpine311_container-x64-1, test-digitalocean-alpine312_container-x64-2, test-rackspace-debian10-x64-1, test-digitalocean-fedora32-x64-1, test-digitalocean-fedora34-x64-1, test-ibm-rhel8-x64-1, test-digitalocean-ubuntu1804-x64-1, test-osuosl-rhel8-ppc64_le-2, test-equinix-rhel8_container-arm64-1, test-equinix-ubuntu1804_container-arm64-2, test-equinix-ubuntu2004_container-arm64-2, test-equinix-ubuntu2004_container-armv7l-1, test-ibm-rhel8-s390x-4, test-equinix-ubuntu1804_sharedlibs_container-arm64-3, test-softlayer-ubi81_container-x64-1, test-softlayer-ubuntu1804_sharedlibs_container-x64-1, test-digitalocean-ubuntu1804_sharedlibs_container-x64-6, test-softlayer-ubuntu1804_sharedlibs_container-x64-4, test-softlayer-ubuntu1804_sharedlibs_container-x64-2, test-digitalocean-ubuntu1804_sharedlibs_container-x64-4, test-digitalocean-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-8, undefined, test-equinix-debian10_container-armv7l-2, test-equinix-ubuntu2004_container-armv7l-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47363/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47408/
Example
not ok 996 parallel/test-fs-promises-watch
  ---
  duration_ms: 0.139
  severity: fail
  exitcode: 1
  stack: |-
    node:internal/process/promises:289
                triggerUncaughtException(err, true /* fromPromise */);
                ^
    
    [AssertionError [ERR_ASSERTION]: Missing expected rejection.] {
      generatedMessage: false,
      code: 'ERR_ASSERTION',
      actual: undefined,
      expected: { code: 'ERR_INVALID_ARG_TYPE' },
      operator: 'rejects'
    }
    
    Node.js v20.0.0-pre
  ...


Reason parallel/test-tls-ticket-cluster
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45060/, https://github.com/nodejs/node/pull/43525/)
Appeared test-nearform-macos10.15-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/47359/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47407/
Example
not ok 2631 parallel/test-tls-ticket-cluster
  ---
  duration_ms: 0.580
  severity: fail
  exitcode: 1
  stack: |-
    [primary] got "listening"
    [primary] connecting 52414 session? false
    [worker] connection reused? false
    [primary] got "not-reused"
    [primary] connecting 52414 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 52414 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 52414 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 52414 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 52414 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 52414 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 52414 session? true
    [worker] connection reused? true
    [primary] got "reused"
 ...

Reason sequential/test-cpu-prof-name
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45058/, https://github.com/nodejs/node/pull/45116/)
Appeared test-equinix-ubuntu1804_sharedlibs_container-arm64-2, test-equinix-ubuntu1804_sharedlibs_container-arm64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/47354/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47406/
Example
not ok 3710 sequential/test-cpu-prof-name
  ---
  duration_ms: 1.117
  severity: fail
  exitcode: 1
  stack: |-
    Dispatching message { "id": 1, "method": "Profiler.enable" }
    Receive CPU profile message
    {"id":1,"result":{}}
    Dispatching message { "id": 2, "method": "Profiler.setSamplingInterval", "params": { "interval": 50 } }
    Receive CPU profile message
    {"id":2,"result":{}}
    Dispatching message { "id": 3, "method": "Profiler.start" }
    Receive CPU profile message
    {"id":3,"result":{}}
    
    
    #
    # Fatal error in ../deps/v8/src/execution/frames.h, line 197
    # Debug check failed: static_cast<uintptr_t>(type) < Type::NUMBER_OF_TYPES (140736983195888 vs. 27).
    #
    #
    #
    #FailureMessage Object: 0xffffc3c7a078
     1: 0xaaaac835c318 node::DumpBacktrace(_IO_FILE*) [/home/iojs/build/workspace/node-test-commit-arm-debug/out/Debug/node]
     2: 0xaaaac853ee28  [/home/iojs/build/workspace/node-test-commit-arm-debug/out/Debug/node]
     3: 0xaaaac853ee5c  [/home/iojs/b...

Jenkins Failure
Reason Failed to create a temp file on /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64
Type JENKINS_FAILURE
Failed PR 9 (https://github.com/nodejs/node/pull/45100/, https://github.com/nodejs/node/pull/45098/, https://github.com/nodejs/node/pull/45119/, https://github.com/nodejs/node/pull/45092/, https://github.com/nodejs/node/pull/45124/, https://github.com/nodejs/node/pull/44713/, https://github.com/nodejs/node/pull/44935/, https://github.com/nodejs/node/pull/45068/, https://github.com/nodejs/node/pull/43525/)
Appeared test-orka-macos11-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47382/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47396/
Example
java.io.IOException: Failed to create a temp file on /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64
	at hudson.FilePath.createTextTempFile(FilePath.java:1626)
	at hudson.tasks.CommandInterpreter.createScriptFile(CommandInterpreter.java:202)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:120)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
	at hudson.plugins.postbuildtask.PostbuildTask.perform(PostbuildTask.java:123)

Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/45091/, https://github.com/nodejs/node/pull/45092/)
Appeared test-digitalocean-freebsd12-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47375/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47416/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 107.170.28.213/107.170.28.213:34155' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
	at jdk.proxy2/jdk.proxy2.$Proxy87.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207)

Build Failure
Reason Error: invalid syntax
Type BUILD_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/45126/, https://github.com/nodejs/node/pull/43525/, https://github.com/nodejs/node/pull/45098/)
Appeared test-rackspace-centos7-x64-1, test-softlayer-debian9-x64-1, test-rackspace-ubuntu1604-x64-2, test-osuosl-centos7-ppc64_le-1, test-equinix-centos7_container-arm64-1, test-ibm-rhel7-s390x-3, test-ibm-aix71-ppc64_be-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-2, test-orka-macos10.14-x64-2, test-orka-macos10.15-x64-1, test-digitalocean-alpine311_container-x64-1, test-rackspace-debian10-x64-1, test-digitalocean-fedora32-x64-1, test-ibm-rhel8-x64-2, test-ibm-ubuntu1804-x64-2, test-digitalocean-ubuntu1804_sharedlibs_container-x64-1, test-softlayer-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-10, test-digitalocean-ubuntu1804_sharedlibs_container-x64-7, test-softlayer-ubuntu1804_sharedlibs_container-x64-5, test-digitalocean-ubuntu1804_sharedlibs_container-x64-5, test-equinix-ubuntu1804_sharedlibs_container-arm64-5, test-ibm-rhel8-s390x-1, test-osuosl-rhel8-ppc64_le-1, undefined
First CI https://ci.nodejs.org/job/node-test-pull-request/47405/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47408/
Example
Error: invalid syntax
make[1]: *** [Makefile:511: test-ci] Error 1
make: *** [Makefile:540: run-ci] Error 2
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True

Reason error: remote GitHub already exists.
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/43525/, https://github.com/nodejs/node/pull/45124/)
Appeared test-rackspace-win2012r2_vs2019-x64-6, test-rackspace-win2012r2_vs2019-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/47391/
Last CI https://ci.nodejs.org/job/node-test-pull-request/47392/
Example
error: remote GitHub already exists.

> git fetch --prune --no-tags GitHub 

> git fetch --prune --no-tags https://github.com/nodejs/build.git +refs/heads/main:refs/remotes/jenkins_tmp

Progress
  • pummel/test-fs-readfile-tostring-fail (17)
  • pummel/test-fs-largefile (5)
  • parallel/test-macos-app-sandbox (4)
  • abort/test-abort-backtrace (2)
  • addons/request-interrupt/test (2)
  • node-api/test_fatal/test2 (2)
  • parallel/test-fs-promises-watch (2)
  • parallel/test-tls-ticket-cluster (2)
  • sequential/test-cpu-prof-name (2)
  • Failed to create a temp file on /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64 (9)
  • Backing channel 'JNLP4-connect connection from ... is disconnected. (2)
  • Error: invalid syntax (3)
  • error: remote GitHub already exists. (2)
Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/reliability

All issues in nodejs/reliability

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.