CI Reliability 2023-05-22

Open
#573 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
20/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
node.js, shell

Research direction

Start with the generated CI report and its linked Jenkins logs, then inspect the named tests: test/parallel/test-runner-output.js, test/parallel/test-http-remove-connection-header-persists-connection.js, test/parallel/test-runner-watch-mode.js, test/pummel/test-net-throttle.js, and test/sequential/test-worker-prof.js. Reproduce the failures on the listed platforms and determine which, if any, require a code or CI change; done is not defined by the issue.

Written by the indexing model from the issue text.

Description

Failures in node-test-pull-request/51804 to node-test-pull-request/51900 that failed more than 2 PRs
(Generated with ncu-ci walk pr --stats=true --markdown /home/runner/work/reliability/reliability/results.md)

UTC Time RUNNING SUCCESS UNSTABLE ABORTED FAILURE Green Rate
2023-05-22 00:14 1 4 36 4 55 4.21%
CCTest Failure
Jenkins Failure
Reason Build timed out (after 60 minutes). Marking the build as failed.
Type JENKINS_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/48034/, https://github.com/nodejs/node/pull/48062/)
Appeared test-digitalocean-ubi81_container-x64-1, test-softlayer-ubi81_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/51852/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51867/
Example
Build timed out (after 60 minutes). Marking the build as failed.

Git Failure
Build Failure
Reason ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Type BUILD_FAILURE
Failed PR 6 (https://github.com/nodejs/node/pull/47939/, https://github.com/nodejs/node/pull/47447/, https://github.com/nodejs/node/pull/47997/, https://github.com/nodejs/node/pull/47576/, https://github.com/nodejs/node/pull/47986/, https://github.com/nodejs/node/pull/48092/)
Appeared test-ibm-aix72-ppc64_be-2, test-equinix-rhel8_container-arm64-1, test-equinix-ubuntu1804_sharedlibs_container-arm64-1, test-equinix-ubuntu2004_container-arm64-2, test-equinix-ubuntu2004_sharedlibs_container-arm64-5, test-equinix_mnx-smartos20-x64-3, test-osuosl-aix72-ppc64_be-3, test-equinix-ubuntu2004_container-armv7l-2, test-equinix-ubuntu2004_container-armv7l-1, test-osuosl-ubuntu2004_container-armv7l-1, test-equinix-debian10_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/51806/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51891/
Example
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Sending e-mails to: michael_dawson@ca.ibm.com vieuxtech@gmail.com ashley.cripps@ibm.com
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: FAILURE

JSTest Failure
Reason parallel/test-runner-output
Type JS_TEST_FAILURE
Failed PR 5 (https://github.com/nodejs/node/pull/48009/, https://github.com/nodejs/node/pull/48033/, https://github.com/nodejs/node/pull/48034/, https://github.com/nodejs/node/pull/47866/, https://github.com/nodejs/node/pull/48038/)
Appeared test-digitalocean-freebsd12-x64-1, test-digitalocean-freebsd12-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/51847/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51898/
Example
not ok 2427 parallel/test-runner-output
  ---
  duration_ms: 13294.75000
  severity: fail
  exitcode: 1
  stack: |-
    TAP version 13
    # Subtest: test runner output
        # Subtest: test-runner/output/abort.js
        ok 1 - test-runner/output/abort.js
          ---
          duration_ms: 12706.747848
          ...


Reason parallel/test-http-remove-connection-header-persists-connection
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/47866/, https://github.com/nodejs/node/pull/47824/, https://github.com/nodejs/node/pull/47986/, https://github.com/nodejs/node/pull/48092/)
Appeared test-softlayer-ubuntu1804_sharedlibs_container-x64-3, test-ibm-rhel8-s390x-3, test-orka-macos11-x64-2, test-orka-macos11-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/51874/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51891/
Example
not ok 1482 parallel/test-http-remove-connection-header-persists-connection
  ---
  duration_ms: 296.79700
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:125
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    false !== true
    
        at /home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-http-remove-connection-header-persists-connection.js:67:16
        at Timeout._onTimeout (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-http-remove-connection-header-persists-connection.js:54:7)
        at listOnTimeout (node:internal/timers:573:17)
        at process.processTimers (node:internal/timers:514:7) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: 'strictEqual'
    }
    
    Node.js v21.0.0-pre
  ...


Reason parallel/test-runner-watch-mode
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/47867/, https://github.com/nodejs/node/pull/48036/, https://github.com/nodejs/node/pull/48078/, https://github.com/nodejs/node/pull/48091/)
Appeared test-rackspace-win2012r2_vs2015-x64-2, test-rackspace-win2012r2_vs2017-x64-4, test-equinix-ubuntu2004_container-arm64-2, test-digitalocean-alpine311_container-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/51805/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51895/
Example
not ok 887 parallel/test-runner-watch-mode
  ---
  duration_ms: 120130.98700
  severity: fail
  exitcode: 1
  stack: |-
    timeout
    TAP version 13
    # Subtest: test runner watch mode
        # Subtest: should run tests repeatedly
        ok 1 - should run tests repeatedly
          ---
          duration_ms: 482.893568
          ...


Reason pummel/test-net-throttle
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/48020/, https://github.com/nodejs/node/pull/47821/, https://github.com/nodejs/node/pull/48092/, https://github.com/nodejs/node/pull/48098/)
Appeared test-digitalocean-freebsd12-x64-1, test-digitalocean-freebsd12-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/51818/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51896/
Example
not ok 3587 pummel/test-net-throttle
  ---
  duration_ms: 442.04200
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:125
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    false !== true
    
        at process.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd12-x64/test/pummel/test-net-throttle.js:87:10)
        at process.emit (node:events:523:35) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: false,
      expected: true,
      operator: 'strictEqual'
    }
    
    Node.js v21.0.0-pre
  ...


Reason sequential/test-worker-prof
Type JS_TEST_FAILURE
Failed PR 4 (https://github.com/nodejs/node/pull/47919/, https://github.com/nodejs/node/pull/47576/, https://github.com/nodejs/node/pull/48078/, https://github.com/nodejs/node/pull/48038/)
Appeared test-azure_msft-win10_vs2019-x64-1, test-rackspace-win2012r2_vs2019-x64-4, test-azure_msft-win10_vs2019-x64-4, test-nearform-macos11.0-arm64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/51837/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51899/
Example
not ok 927 sequential/test-worker-prof
  ---
  duration_ms: 1564.73100
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:125
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: child exited with non-zero status:                      {
      status: 3221226505,
      signal: null,
      output: [ null, '', '' ],
      pid: 1068,
      stdout: '',
      stderr: ''
    }
        at Object.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\sequential\test-worker-prof.js:61:10)
        at Module._compile (node:internal/modules/cjs/loader:1255:14)
        at Module._extensions..js (node:internal/modules/cjs/loader:1309:10)
        at Module.load (node:internal/modules/cjs/loader:1113:32)
        at Module._load (node:internal/modules/cjs/loader:960:12)
        at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
        at node:internal/main/run_main_module:23:47 {
      generatedMessage: false,
      code: 'ERR_ASSERTIO...

Reason js-native-api/test_cannot_run_js/test
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/47986/, https://github.com/nodejs/node/pull/48098/, https://github.com/nodejs/node/pull/48038/)
Appeared test-equinix-ubuntu2004_container-armv7l-2, test-digitalocean-freebsd12-x64-2, test-nearform-macos10.15-x64-3, test-softlayer-debian10-x64-1, test-digitalocean-fedora32-x64-1, test-digitalocean-fedora34-x64-2, test-ibm-rhel8-x64-1, test-digitalocean-ubuntu1804-x64-1, test-ibm-rhel8-s390x-3, test-osuosl-ubuntu2004_container-armv7l-1
First CI https://ci.nodejs.org/job/node-test-pull-request/51885/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51898/
Example
not ok 86 js-native-api/test_cannot_run_js/test
  ---
  duration_ms: 514.38900
  severity: crashed
  exitcode: -6
  stack: |-
  ...


Reason parallel/test-single-executable-application
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/47433/, https://github.com/nodejs/node/pull/47447/, https://github.com/nodejs/node/pull/47455/)
Appeared test-orka-macos11-x64-2, test-orka-macos11-x64-1, test-digitalocean-ubuntu1804_sharedlibs_container-x64-7, test-digitalocean-ubuntu1804_sharedlibs_container-x64-5, test-digitalocean-ubuntu1804_sharedlibs_container-x64-8
First CI https://ci.nodejs.org/job/node-test-pull-request/51839/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51842/
Example
not ok 2755 parallel/test-single-executable-application
  ---
  duration_ms: 120.88
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason sequential/test-inspector-debug-end
Type JS_TEST_FAILURE
Failed PR 3 (https://github.com/nodejs/node/pull/47821/, https://github.com/nodejs/node/pull/48034/, https://github.com/nodejs/node/pull/48033/)
Appeared test-ibm-rhel7-s390x-3
First CI https://ci.nodejs.org/job/node-test-pull-request/51850/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51853/
Example
not ok 3690 sequential/test-inspector-debug-end
  ---
  duration_ms: 120.250
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    Test there's no crash stopping server that was not started
    Test there's no crash stopping server without connecting
    [err] Debugger listening on ws://127.0.0.1:36322/d4d6b8ed-50c7-4e25-a533-09f9b69698d4
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    Test there's no crash stopping server after connecting
    [test] Connecting to a child Node process
    [test] Testing /json/list
    [err] Debugger listening on ws://127.0.0.1:38574/965d12a2-7dd3-4193-95e3-b43922a3dfdd
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    [err] Debugger attached.
    [err] 
  ...


Reason parallel/test-fs-watch-recursive-symlink
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/48033/, https://github.com/nodejs/node/pull/48072/)
Appeared test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/51853/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51870/
Example
not ok 1328 parallel/test-fs-watch-recursive-symlink
  ---
  duration_ms: 120027.42000
  severity: fail
  exitcode: -15
  stack: |-
    timeout
  ...


Reason parallel/test-http-request-join-authorization-headers
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/47433/, https://github.com/nodejs/node/pull/47447/)
Appeared test-softlayer-ubuntu1804_sharedlibs_container-x64-4, test-digitalocean-ubuntu1804_sharedlibs_container-x64-7, test-softlayer-ubuntu1804_sharedlibs_container-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/51839/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51841/
Example
not ok 1433 parallel/test-http-request-join-authorization-headers
  ---
  duration_ms: 0.303
  severity: fail
  exitcode: 1
  stack: |-
    node:assert:125
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    
    '1, 2' !== '1'
    
        at Server.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-http-request-join-authorization-headers.js:39:12)
        at Server.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/common/index.js:447:15)
        at Server.emit (node:events:513:28)
        at parserOnIncoming (node:_http_server:1091:12)
        at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: '1, 2',
      expected: '1',
      operator: 'strictEqual'
    }
    
    Node.js v18.16.1-pre
  ...


Reason parallel/test-inspector-console
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/47821/, https://github.com/nodejs/node/pull/47866/)
Appeared test-ibm-rhel8-s390x-4
First CI https://ci.nodejs.org/job/node-test-pull-request/51850/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51886/
Example
not ok 1781 parallel/test-inspector-console
  ---
  duration_ms: 15591.77600
  severity: fail
  exitcode: 1
  stack: |-
    [test] Connecting to a child Node process
    [test] Testing /json/list
    [err] Debugger listening on ws://127.0.0.1:44615/736cb216-9fc5-4a21-b730-1e68e6fc004b
    [err] For help, see: https://nodejs.org/en/docs/inspector
    [err] 
    [err] Debugger attached.
    [err] 
    Timed out waiting for matching notification (Runtime.consoleAPICalled)
    1
  ...


Reason parallel/test-repl-preview
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/47336/, https://github.com/nodejs/node/pull/47821/)
Appeared test-ibm-rhel7-s390x-3
First CI https://ci.nodejs.org/job/node-test-pull-request/51820/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51822/
Example
not ok 2348 parallel/test-repl-preview
  ---
  duration_ms: 2.10
  severity: fail
  exitcode: 1
  stack: |-
    Testing foo
    Testing foo
    Testing foo
    Testing foo
    Testing foo
    Testing koo
    Testing koo
    Testing koo
    Testing a
    Testing a
    Testing a
    Testing  { b: 1 }['b'] === 1
    Testing  { b: 1 }['b'] === 1
    Testing  { b: 1 }['b'] === 1
    Testing { b: 1 }['b'] === 1;
    Testing { b: 1 }['b'] === 1;
    Testing { b: 1 }['b'] === 1;
    Testing koo
    Testing koo
    Testing { a: true }
    Testing { a: true }
    Testing { a: true }
    Testing { a: true };
    Testing { a: true };
    Testing { a: true };
    Testing  	 { a: true};
    Testing  	 { a: true};
    Testing  	 { a: true};
    Testing 1n + 2n
    Testing 1n + 2n
    Testing 1n + 2n
    Testing {};1
    Testing {};1
    Testing {};1
    Testing a
    Testing a
    Testing  { b: 1 }['b'] === 1
    Testing  { b: 1 }['b'] === 1
    Testing { b: 1 }['b'] === 1;
    Testing { b: 1 }['b'] === 1;
    Testing { a: ...

Reason parallel/test-tls-ticket-cluster
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/47821/, https://github.com/nodejs/node/pull/47866/)
Appeared test-azure_msft-win10_vs2019-x64-2
First CI https://ci.nodejs.org/job/node-test-pull-request/51851/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51883/
Example
not ok 754 parallel/test-tls-ticket-cluster
  ---
  duration_ms: 525.00000
  severity: fail
  exitcode: 1
  stack: |-
    [primary] got "listening"
    [primary] connecting 60990 session? false
    [primary] got "listening"
    [primary] got "listening"
    [primary] got "listening"
    [primary] connecting 60990 session? false
    [worker] connection reused? false
    [primary] got "not-reused"
    [primary] connecting 60990 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 60990 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 60990 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 60990 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 60990 session? true
    [worker] connection reused? true
    [primary] got "reused"
    [primary] connecting 60990 session? true
    [worker] connection reused? tru...

Reason wpt/test-webcrypto
Type JS_TEST_FAILURE
Failed PR 2 (https://github.com/nodejs/node/pull/48024/, https://github.com/nodejs/node/pull/48038/)
Appeared test-orka-macos11-x64-1, test-softlayer-alpine311_container-x64-1, test-digitalocean-alpine312_container-x64-2, test-equinix-ubuntu2004_sharedlibs_container-arm64-5, test-equinix-rhel8_container-arm64-1, test-equinix-ubuntu2004_container-armv7l-1, test-azure_msft-win11_vs2022-arm64-1, test-rackspace-win2012r2_vs2017-x64-3
First CI https://ci.nodejs.org/job/node-test-pull-request/51821/
Last CI https://ci.nodejs.org/job/node-test-pull-request/51876/
Example
not ok 3943 wpt/test-webcrypto
  ---
  duration_ms: 67135.04800
  severity: crashed
  exitcode: -6
  stack: |-
    [SKIPPED] algorithm-discards-context.https.window.js: Not relevant in Node.js context
    [SKIPPED] historical.any.js: Not relevant in Node.js context
    [PASS] setup - define tests
    [PASS] X25519 key derivation checks for all-zero value result with a key of order 0
    [PASS] X25519 key derivation checks for all-zero value result with a key of order 1
    [PASS] X25519 key derivation checks for all-zero value result with a key of order 8
    [PASS] X25519 key derivation checks for all-zero value result with a key of order p-1 (order 2)
    [PASS] X25519 key derivation checks for all-zero value result with a key of order p (=0, order 4)
    [PASS] X25519 key derivation checks for all-zero value result with a key of order p+1 (=1, order 1)
    [PASS] X448 key derivation checks for all-zero value result with a key of order 0
    [PASS] X448 key derivation checks for all-zero value result with a...

Progress
  • Build timed out (after 60 minutes). Marking the build as failed. (2)
  • ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? (6)
  • parallel/test-runner-output (5)
  • parallel/test-http-remove-connection-header-persists-connection (4)
  • parallel/test-runner-watch-mode (4)
  • pummel/test-net-throttle (4)
  • sequential/test-worker-prof (4)
  • js-native-api/test_cannot_run_js/test (3)
  • parallel/test-single-executable-application (3)
  • sequential/test-inspector-debug-end (3)
  • parallel/test-fs-watch-recursive-symlink (2)
  • parallel/test-http-request-join-authorization-headers (2)
  • parallel/test-inspector-console (2)
  • parallel/test-repl-preview (2)
  • parallel/test-tls-ticket-cluster (2)
  • wpt/test-webcrypto (2)
Dominant language
Shell
Stars
57
Forks
20
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nodejs/reliability

All issues in nodejs/reliability

Similar issues

More Shell/Bash issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.