CI Reliability 2021-12-27

Open
#157 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
node.js, shell
Domain
ci-cd, testing-qa

Research direction

Start with the generated ncu-ci walk pr --stats=true --markdown report and reproduce the listed failures, including parallel/test-fs-rmdir-recursive, parallel/test-repl-sigint-nested-eval, and sequential/test-performance-eventloopdelay. Review the referenced CI logs and test files; done would require identifying and addressing the reliability causes so these failures no longer repeatedly break pull-request CI.

Written by the indexing model from the issue text.

Description

Failures in node-test-pull-request/41563 to node-test-pull-request/41649 that failed more than 2 PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2021-12-27 00:01 7 10 18 11 54 12.20%
JSTest Failure
Reason parallel/test-fs-rmdir-recursive
Type JS_TEST_FAILURE
Failed PR 8 (https://github.com/nodejs/node/pull/41185/, https://github.com/nodejs/node/pull/41253/, https://github.com/nodejs/node/pull/41271/, https://github.com/nodejs/node/pull/41279/, https://github.com/nodejs/node/pull/41291/, https://github.com/nodejs/node/pull/41288/, https://github.com/nodejs/node/pull/41305/, https://github.com/nodejs/node/pull/38590/)
Appeared test-azure_msft-win10_vs2019-x64-4, test-azure_msft-win10_vs2019-x64-3, test-azure_msft-win10_vs2019-x64-2, test-azure_msft-win10_vs2019-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/41574/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41644/
Example
not ok 261 parallel/test-fs-rmdir-recursive
  ---
  duration_ms: 0.824
  severity: fail
  exitcode: 1
  stack: |-
    (node:10756) [DEP0147] DeprecationWarning: In future versions of Node.js, fs.rmdir(path, { recursive: true }) will be removed. Use fs.rm(path, { recursive: true }) instead
    (Use `node --trace-deprecation ...` to show where the warning was created)
    Can't clean tmpdir: C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.258
    Files blocking: [ 'rmdir-recursive-0', 'rmdir-recursive-4', 'rmdir-recursive-5' ]
    
    node:assert:991
        throw newErr;
        ^
    
    AssertionError [ERR_ASSERTION]: ifError got unwanted exception: ENOTEMPTY: directory not empty, rmdir 'C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.258\rmdir-recursive-0\d-3-2\d-2-1'
        at C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:364:12
        at C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:401:15
        at node:fs:188:23
 ...

Reason parallel/test-repl-sigint-nested-eval
Type JS_TEST_FAILURE
Failed PR 8 (https://github.com/nodejs/node/pull/41202/, https://github.com/nodejs/node/pull/41185/, https://github.com/nodejs/node/pull/41246/, https://github.com/nodejs/node/pull/41266/, https://github.com/nodejs/node/pull/40532/, https://github.com/nodejs/node/pull/41218/, https://github.com/nodejs/node/pull/41291/, https://github.com/nodejs/node/pull/41221/)
Appeared test-softlayer-ubuntu1804_sharedlibs_container-x64-2, test-digitalocean-ubuntu1804_sharedlibs_container-x64-5, test-digitalocean-ubuntu1804_sharedlibs_container-x64-4, test-digitalocean-ubuntu1804_sharedlibs_container-x64-7, test-digitalocean-alpine312_container-x64-2, test-osuosl-aix72-ppc64_be-4, test-digitalocean-ubuntu1804_sharedlibs_container-x64-6, test-softlayer-ubuntu1804_sharedlibs_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/41567/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41633/
Example
not ok 2184 parallel/test-repl-sigint-nested-eval
  ---
  duration_ms: 0.539
  severity: fail
  exitcode: 1
  stack: |-
    Stacktrace:
       ptr1=0x178a08f01d49
        ptr2=(nil)
        ptr3=(nil)
        ptr4=(nil)
        failure_message_object=0x7ffff8edb2b0
    
    ==== JS stack trace =========================================
    
        0: ExitFrame [pc: 0x55e8894e73f9]
        1: StubFrame [pc: 0x55e88957145d]
    Security context: 0x32eb29db3be1 <JSGlobalObject>
        2: _fatalException [0x32eb29d89159] [node:internal/process/execution:145] [bytecode=0x16c7878a2721 offset=13](this=0x32eb29d82709 <process map = 0x196788d81479>,0x178a08f01d49 <Odd Oddball: termination_exception>,0x178a08f01769 <false>)
        3: InternalFrame [pc: 0x55e88947465b]
        4: EntryFrame [pc: 0x55e889474383]
        5: builtin exit frame: evaluate(this=0x161e859a16c9 <ModuleWrap map = 0x19af5bf8b631>,0x161e859a16c9 <ModuleWrap map = 0x19af5bf8b631>,-1,0x178a08f01769 <false>)
    
        6: getESMFacade [0x3445e35d...

Reason sequential/test-performance-eventloopdelay
Type JS_TEST_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/41185/, https://github.com/nodejs/node/pull/41240/, https://github.com/nodejs/node/pull/41203/, https://github.com/nodejs/node/pull/41279/, https://github.com/nodejs/node/pull/41280/, https://github.com/nodejs/node/pull/41221/)
Appeared test-ibm-rhel7-s390x-1, test-ibm-rhel7-s390x-4, test-ibm-rhel7-s390x-3
First CI https://ci.nodejs.org/job/node-test-pull-request/41581/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41649/
Example
not ok 3533 sequential/test-performance-eventloopdelay
  ---
  duration_ms: 7.224
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:399
        throw err;
        ^
    
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
    
      assert(histogram.min > 0)
    
        at Timeout.spinAWhile [as _onTimeout] (/home/iojs/build/workspace/node-test-commit-linuxone/test/sequential/test-performance-eventloopdelay.js:65:7)
        at listOnTimeout (node:internal/timers:568:17)
        at processTimers (node:internal/timers:510:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: '=='
    }
    
    Node.js v18.0.0-pre
  ...


Reason parallel/test-crypto-dh-leak
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/41253/, https://github.com/nodejs/node/pull/41218/, https://github.com/nodejs/node/pull/41271/, https://github.com/nodejs/node/pull/41307/)
Appeared test-requireio_svincent-debian10-armv7l_pi2-3, test-requireio_joeyvandijk-debian10-armv7l_pi2-2, test-requireio_securogroup-debian10-arm64_pi3-1
First CI https://ci.nodejs.org/job/node-test-pull-request/41575/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41643/
Example
not ok 159 parallel/test-crypto-dh-leak
  ---
  duration_ms: 240.279
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-trace-events-fs-sync
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/41240/, https://github.com/nodejs/node/pull/41203/, https://github.com/nodejs/node/pull/41276/, https://github.com/nodejs/node/pull/41302/)
Appeared test-azure_msft-win10_vcbt2015-x64-4, test-azure_msft-win10_vs2019-x64-1, test-azure_msft-win10_vcbt2015-x64-2, test-azure_msft-win10_vs2019-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/41594/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41629/
Example
not ok 699 parallel/test-trace-events-fs-sync
  ---
  duration_ms: 3.913
  severity: fail
  exitcode: 1
  stack: |-
    Can't clean tmpdir: C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.681
    Files blocking: [ 'fs.txt', 'node_trace.1.log' ]
    
    C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-trace-events-fs-sync.js:133
        throw new Error(`${tr}:\n${util.inspect(proc)}`);
        ^
    
    Error: fs.sync.ftruncate:
    {
      status: 1,
      signal: null,
      output: [
        null,
        '',
        'node:internal/fs/utils:344\r\n' +
          '    throw err;\r\n' +
          '    ^\r\n' +
          '\r\n' +
          "Error: EPERM: operation not permitted, open 'fs.txt'\r\n" +
          '    at Object.openSync (node:fs:585:3)\r\n' +
          '    at Object.writeFileSync (node:fs:2155:35)\r\n' +
          '    at [eval]:1:4\r\n' +
          '    at Script.runInThisContext (node:vm:129:12)\r\n' +
          '    at Object.runInThisContext (node:vm:305...

Reason wpt/test-user-timing
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/41253/, https://github.com/nodejs/node/pull/41240/, https://github.com/nodejs/node/pull/41266/, https://github.com/nodejs/node/pull/41279/)
Appeared test-requireio_mininodes-debian10-armv7l_pi2-1, test-requireio_pivotalagency-debian10-arm64_pi3-2, test-requireio_ceejbot-debian10-armv7l_pi2-1
First CI https://ci.nodejs.org/job/node-test-pull-request/41575/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41613/
Example
not ok 854 wpt/test-user-timing
  ---
  duration_ms: 9.510
  severity: fail
  exitcode: 1
  stack: |-
    ---- idlharness.any.js ----
    [SKIPPED] idlharness cannot recognize Node.js environment
    ---- invoke_with_timing_attributes.worker.js ----
    [SKIPPED] importScripts not supported
    ---- performance-measure-invalid.worker.js ----
    [SKIPPED] importScripts not supported
    ---- buffered-flag.any.js ----
    [PASS] PerformanceObserver with buffered flag sees previous marks
    ---- buffered-flag.any.js ----
    [PASS] PerformanceObserver with buffered flag sees previous measures
    ---- case-sensitivity.any.js ----
    [PASS] getEntriesByType values are case sensitive
    ---- clear_all_marks.any.js ----
    [PASS] Clearing all marks remove all of them.
    ---- clear_one_mark.any.js ----
    [PASS] Clearing an existent mark doesn't affect other existing marks
    ---- clear_non_existent_mark.any.js ----
    [PASS] Clearing a non-existent mark doesn't affect existing marks
    ---- entry_type.an...

Reason parallel/test-crypto-binary-default
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/40901/, https://github.com/nodejs/node/pull/41240/, https://github.com/nodejs/node/pull/41290/)
Appeared test-requireio_joeyvandijk-debian10-armv7l_pi2-2, test-requireio_rvagg-debian10-armv7l_pi2-1, test-requireio_davglass-debian10-arm64_pi3-1
First CI https://ci.nodejs.org/job/node-test-pull-request/41564/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41635/
Example
not ok 209 parallel/test-crypto-binary-default
  ---
  duration_ms: 240.136
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-crypto-dh
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/41279/, https://github.com/nodejs/node/pull/41302/, https://github.com/nodejs/node/pull/41307/)
Appeared test-requireio_ceejbot-debian10-armv7l_pi2-1, test-requireio_mininodes-debian10-armv7l_pi2-1
First CI https://ci.nodejs.org/job/node-test-pull-request/41613/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41643/
Example
not ok 222 parallel/test-crypto-dh
  ---
  duration_ms: 240.593
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-fs-rm
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/41203/, https://github.com/nodejs/node/pull/41257/, https://github.com/nodejs/node/pull/41246/)
Appeared test-azure_msft-win10_vs2019-x64-1, test-azure_msft-win10_vs2019-x64-4
First CI https://ci.nodejs.org/job/node-test-pull-request/41577/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41589/
Example
not ok 257 parallel/test-fs-rm
  ---
  duration_ms: 0.550
  severity: fail
  exitcode: 1
  stack: |-
    node:internal/fs/rimraf:202
          throw err;
          ^
    
    Error: ENOTEMPTY: directory not empty, rmdir '\\?\C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.255\rm-4\d-3-1'
        at Object.rmdirSync (node:fs:1167:10)
        at _rmdirSync (node:internal/fs/rimraf:260:21)
        at rimrafSync (node:internal/fs/rimraf:193:7)
        at node:internal/fs/rimraf:253:9
        at Array.forEach (<anonymous>)
        at _rmdirSync (node:internal/fs/rimraf:250:7)
        at rimrafSync (node:internal/fs/rimraf:193:7)
        at Object.rmSync (node:fs:1216:10)
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\parallel\test-fs-rm.js:176:6)
        at Module._compile (node:internal/modules/cjs/loader:1097:14) {
      errno: -4051,
      syscall: 'rmdir',
      code: 'ENOTEMPTY',
      path: '\\\\?\\C:\\workspace\\node-test-binary-windows-js-suites\\nod...

Reason parallel/test-crypto-dh-odd-key
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/41302/, https://github.com/nodejs/node/pull/38590/)
Appeared test-requireio_ceejbot-debian10-armv7l_pi2-1, test-requireio_mcollina-debian10-armv7l_pi2-1
First CI https://ci.nodejs.org/job/node-test-pull-request/41632/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41644/
Example
not ok 239 parallel/test-crypto-dh-odd-key
  ---
  duration_ms: 240.114
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason pummel/test-worker-take-heapsnapshot
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/41263/, https://github.com/nodejs/node/pull/41305/)
Appeared test-equinix-ubuntu2004_sharedlibs_container-arm64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/41630/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41638/
Example
not ok 3216 pummel/test-worker-take-heapsnapshot
  ---
  duration_ms: 0.513
  severity: crashed
  exitcode: -11
  stack: |-
    (node:989228) internal/test/binding: These APIs are for internal testing only. Do not use them.
    (Use `node --trace-warnings ...` to show where the warning was created)
  ...


Jenkins Failure
Reason Backing channel 'JNLP4-connect connection from ... is disconnected.
Type JENKINS_FAILURE
Failed PR 7 (https://github.com/nodejs/node/pull/41240/, https://github.com/nodejs/node/pull/41271/, https://github.com/nodejs/node/pull/41253/, https://github.com/nodejs/node/pull/41279/, https://github.com/nodejs/node/pull/41302/, https://github.com/nodejs/node/pull/41263/, https://github.com/nodejs/node/pull/41236/)
Appeared test-equinix-ubuntu2004_container-armv7l-1, test-equinix-centos7_container-arm64-2, test-equinix-ubuntu2004_container-arm64-2, test-osuosl-aix72-ppc64_be-3, test-osuosl-aix72-ppc64_be-2
First CI https://ci.nodejs.org/job/node-test-pull-request/41576/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41647/
Example
java.io.IOException: Backing channel 'JNLP4-connect connection from 145.40.81.219/145.40.81.219:45472' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:216)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
	at com.sun.proxy.$Proxy82.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1213)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1205)

Reason Build timed out (after 60 minutes). Marking the build as failed.
Type JENKINS_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/41246/, https://github.com/nodejs/node/pull/40532/, https://github.com/nodejs/node/pull/41291/, https://github.com/nodejs/node/pull/41306/)
Appeared test-softlayer-ubi81_container-x64-1, test-digitalocean-ubi81_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/41602/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41639/
Example
Build timed out (after 60 minutes). Marking the build as failed.

Build Failure
Reason fatal error: ld terminated with signal 9 [Killed]
Type BUILD_FAILURE
Failed PR 10 (https://github.com/nodejs/node/pull/41185/, https://github.com/nodejs/node/pull/41253/, https://github.com/nodejs/node/pull/41240/, https://github.com/nodejs/node/pull/41270/, https://github.com/nodejs/node/pull/41271/, https://github.com/nodejs/node/pull/39239/, https://github.com/nodejs/node/pull/41235/, https://github.com/nodejs/node/pull/41276/, https://github.com/nodejs/node/pull/41218/, https://github.com/nodejs/node/pull/41246/)
Appeared test-digitalocean-ubuntu1804_sharedlibs_container-x64-2, test-digitalocean-ubuntu1804_sharedlibs_container-x64-4, test-digitalocean-ubuntu1804_sharedlibs_container-x64-3, test-digitalocean-ubuntu1804_sharedlibs_container-x64-5, test-digitalocean-ubuntu1804_sharedlibs_container-x64-8
First CI https://ci.nodejs.org/job/node-test-pull-request/41574/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41609/
Example
fatal error: ld terminated with signal 9 [Killed]

Reason ERROR: Step ?Publish JUnit test result report? failed: no workspace for ...
Type BUILD_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/41202/, https://github.com/nodejs/node/pull/41253/, https://github.com/nodejs/node/pull/41263/, https://github.com/nodejs/node/pull/41305/)
Appeared test-equinix-centos7_container-arm64-2, test-equinix-ubuntu1804_container-arm64-2, test-equinix-ubuntu2004_container-arm64-2, test-osuosl-aix72-ppc64_be-4, test-osuosl-aix72-ppc64_be-2
First CI https://ci.nodejs.org/job/node-test-pull-request/41567/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41642/
Example
ERROR: Step ?Publish JUnit test result report? failed: no workspace for node-test-commit-arm/nodes=centos7-arm64-gcc8 #40060
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

Reason Failed to trigger node-test-commit
Type BUILD_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/41575/, )
Appeared test-packetnet-ubuntu1804-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/41588/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41628/
Example
Failed to trigger node-test-commit

undefined
Reason Unknown
Type undefined
Failed PR 16 (https://github.com/nodejs/node/pull/40901/, https://github.com/nodejs/node/pull/41240/, https://github.com/nodejs/node/pull/41218/, https://github.com/nodejs/node/pull/41246/, https://github.com/nodejs/node/pull/41266/, https://github.com/nodejs/node/pull/41276/, https://github.com/nodejs/node/pull/41279/, https://github.com/nodejs/node/pull/41253/, https://github.com/nodejs/node/pull/41271/, https://github.com/nodejs/node/pull/41263/, https://github.com/nodejs/node/pull/41302/, https://github.com/nodejs/node/pull/41280/, https://github.com/nodejs/node/pull/41290/, https://github.com/nodejs/node/pull/41305/, https://github.com/nodejs/node/pull/41307/, https://github.com/nodejs/node/pull/38590/)
Appeared test-equinix-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/41564/
Last CI https://ci.nodejs.org/job/node-test-pull-request/41644/
Example
Unknown

Progress
  • parallel/test-fs-rmdir-recursive (8)
  • parallel/test-repl-sigint-nested-eval (8)
  • sequential/test-performance-eventloopdelay (6)
  • parallel/test-crypto-dh-leak (4)
  • parallel/test-trace-events-fs-sync (4)
  • wpt/test-user-timing (4)
  • parallel/test-crypto-binary-default (3)
  • parallel/test-crypto-dh (3)
  • parallel/test-fs-rm (3)
  • parallel/test-crypto-dh-odd-key (2)
  • pummel/test-worker-take-heapsnapshot (2)
  • Backing channel 'JNLP4-connect connection from ... is disconnected. (7)
  • Build timed out (after 60 minutes). Marking the build as failed. (4)
  • fatal error: ld terminated with signal 9 [Killed] (10)
  • ERROR: Step ?Publish JUnit test result report? failed: no workspace for ... (4)
  • Failed to trigger node-test-commit (2)
  • Unknown (16)
Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/reliability

All issues in nodejs/reliability

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.