nodejs / nodejs/node-core-utils

`ncu-ci`: output a summary list with all failing tests

Open
#396 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
JavaScript
Stars
314
Forks
137
Avg merge
2d 8h
Merged PRs (30d)
24

Description

So it's easier to know which tests are failing across all platforms. Something like:

------------------------------- Failures Summary -------------------------------
Build failures:
  - win-vs2017
  - cross-compiler-armv7-gcc-6
  - osx1011
Test failures:
  - sequential/test-inspector-break-when-eval (4 platforms)
  - sequential/test-perf-hooks (2 platforms)
  - parallel/test-memory-usage (1 platform)
  - parallel/test-repl-history-navigation (1 platform)
  - sequential/test-heap-prof (1 platform)
  - sequential/test-worker-prof (1 platform)

Example:

$ ncu-ci url ci.nodejs.org/job/node-test-pull-request/29794
--------------------------------------------------------------------------------
[1/1] Running PR: 29597
--------------------------------------------------------------------------------
✔  Build data downloaded
✔  Data downloaded
----------------------------------- Summary ------------------------------------
Result     FAILURE
URL        https://ci.nodejs.org/job/node-test-pull-request/29597/
Source     https://github.com/nodejs/node/pull/32116/
Commit     [1ab2b73b513c] test: warn when inspector process crashes
Date       2020-03-06 16:25:26 -0800
Author     Matheus Marchini <mmarchini@netflix.com>
------------------------------- Failures Summary -------------------------------
Build failures:
  - win-vs2017
  - cross-compiler-armv7-gcc-6
  - osx1011
Test failures:
  - sequential/test-inspector-break-when-eval (4 platforms)
  - sequential/test-perf-hooks (2 platforms)
  - parallel/test-memory-usage (1 platform)
  - parallel/test-repl-history-navigation (1 platform)
  - sequential/test-heap-prof (1 platform)
  - sequential/test-worker-prof (1 platform)
----------------------------------- osx1011 ------------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx1011/32407/console
Type       BUILD_FAILURE
Built On   test-macstadium-macos10.11-x64-1
Reason     fatal: No rebase in progress?
----------------------------------- osx1015 ------------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx1015/32407/console
Type       JS_TEST_FAILURE
Built On   test-nearform-macos10.15-x64-3
Reason
  not ok 2777 sequential/test-inspector-break-when-eval
    ---
    duration_ms: 15.390
    severity: fail
    exitcode: 1
    stack: |-
      [test] Connecting to a child Node process
      [test] Testing /json/list
      [err] Debugger listening on ws://127.0.0.1:63158/870bb3cd-ad48-4622-8a87-53091991bef8
      [err] For help, see: https://nodejs.org/en/docs/inspector
      [err]
      [test] Setting up a debugger
      [err] Debugger attached.
      [err]
      [test] Breaking in the code
      [err] child process crashed, signal SIGSEGV
      Timed out waiting for matching notification (break on file:///Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/test/fixtures/inspector-global-function.js:9))
      1
    ...

------------------------------- ppcle-ubuntu1404 -------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-plinux/nodes=ppcle-ubuntu1404/31338/console
Type       JS_TEST_FAILURE
Built On   test-osuosl-ubuntu1404-ppc64_le-2
Reason
  not ok 2792 sequential/test-inspector-break-when-eval
    ---
    duration_ms: 15.341
    severity: fail
    exitcode: 1
    stack: |-
      [test] Connecting to a child Node process
      [test] Testing /json/list
      [err] Debugger listening on ws://127.0.0.1:52357/f0bedc84-050f-4b9e-a641-286ac55dcb28
      [err] For help, see: https://nodejs.org/en/docs/inspector
      [err]
      [test] Setting up a debugger
      [err] Debugger attached.
      [err]
      [test] Breaking in the code
      [err] child process crashed, signal SIGSEGV
      Timed out waiting for matching notification (break on file:///home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/fixtures/inspector-global-function.js:9))
      1
    ...

--------------------------------- smartos18-64 ---------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-smartos/nodes=smartos18-64/31307/console
Type       JS_TEST_FAILURE
Built On   test-joyent-smartos18-x64-3
Reason
  not ok 2847 sequential/test-inspector-break-when-eval
    ---
    duration_ms: 15.745
    severity: fail
    exitcode: 1
    stack: |-
      [test] Connecting to a child Node process
      [test] Testing /json/list
      [err] Debugger listening on ws://127.0.0.1:44215/d810e836-4d14-4696-a415-2aa7f92714e5
      [err] For help, see: https://nodejs.org/en/docs/inspector
      [err]
      [test] Setting up a debugger
      [err] Debugger attached.
      [err]
      [test] Breaking in the code
      [err] child process crashed, signal SIGSEGV
      Timed out waiting for matching notification (break on file:///home/iojs/build/workspace/node-test-commit-smartos/nodes/smartos18-64/test/fixtures/inspector-global-function.js:9))
      1
    ...

--------------------- ubuntu1804_sharedlibs_withoutssl_x64 ---------------------
URL        https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_withoutssl_x64/18497/console
Type       JS_TEST_FAILURE
Built On   test-joyent-ubuntu1804_sharedlibs_container-x64-1
Reason
  not ok 1393 parallel/test-memory-usage
    ---
    duration_ms: 1.351
    severity: fail
    exitcode: 1
    stack: |-
      assert.js:384
          throw err;
          ^

      AssertionError [ERR_ASSERTION]: 11571790 - 1102879 >= 10485760
          at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-memory-usage.js:42:3)
          at Module._compile (internal/modules/cjs/loader.js:1202:30)
          at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
          at Module.load (internal/modules/cjs/loader.js:1051:32)
          at Function.Module._load (internal/modules/cjs/loader.js:947:14)
          at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
          at internal/main/run_main_module.js:17:47 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: false,
        expected: true,
        operator: '=='
      }
    ...

--------------------- ubuntu1804_sharedlibs_withoutssl_x64 ---------------------
URL        https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_withoutssl_x64/18497/console
Type       JS_TEST_FAILURE
Built On   test-joyent-ubuntu1804_sharedlibs_container-x64-1
Reason
  not ok 2768 sequential/test-perf-hooks
    ---
    duration_ms: 1.83
    severity: fail
    exitcode: 1
    stack: |-
      {
        name: 'node',
        entryType: 'node',
        startTime: 0,
        duration: { around: 468.111141204834 },
        nodeStart: { around: 0 },
        v8Start: { around: 0 },
        bootstrapComplete: { around: 457.42479133605957, delay: 2500 },
        environment: { around: 0 },
        loopStart: -1,
        loopExit: -1
      }
      assert.js:384
          throw err;
          ^

      AssertionError [ERR_ASSERTION]: v8Start: 275.37767028808594 >= 250
          at checkNodeTiming (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-perf-hooks.js:67:7)
          at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-perf-hooks.js:79:1)
          at Module._compile (internal/modules/cjs/loader.js:1202:30)
          at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
          at Module.load (internal/modules/cjs/loader.js:1051:32)
          at Function.Module._load (internal/modules/cjs/loader.js:947:14)
          at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
          at internal/main/run_main_module.js:17:47 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: false,
        expected: true,
        operator: '=='
      }
    ...

------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL        https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type       JS_TEST_FAILURE
Built On   test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
  not ok 1648 parallel/test-repl-history-navigation
    ---
    duration_ms: 12.367
    severity: fail
    exitcode: 1
    stack: |-
      assert.js:102
        throw new AssertionError(obj);
        ^

      AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
      + actual - expected ... Lines skipped

        [
          '\x1B[1G',
      ...

------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL        https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type       JS_TEST_FAILURE
Built On   test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
  not ok 2741 sequential/test-heap-prof
    ---
    duration_ms: 120.338
    severity: fail
    exitcode: -15
    stack: |-
      timeout
    ...

------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL        https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type       JS_TEST_FAILURE
Built On   test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
  not ok 2768 sequential/test-inspector-break-when-eval
    ---
    duration_ms: 6.359
    severity: fail
    exitcode: 1
    stack: |-
      [test] Connecting to a child Node process
      [test] Testing /json/list
      [err] Debugger listening on ws://127.0.0.1:43601/97b597e2-1f2f-470a-b4f6-183159499193
      [err] For help, see: https://nodejs.org/en/docs/inspector
      [err]
      [out] Ready!
      [out]
      [test] Setting up a debugger
      [err] Debugger attached.
      [err]
      [test] Breaking in the code
      [test] Step over console statement and test output
      [out] 0 3
      [out]
      [test] Verify node waits for the frontend to disconnect
      [err] Waiting for the debugger to disconnect...
      [err]
      [err] internal/timers.js:404
      [err]   node.priorityQueuePosition = pos;
      [err]                              ^
      [err]
      [err] TypeError: Cannot set property 'priorityQueuePosition' of undefined
      [err]     at setPosition (internal/timers.js:404:30)
      [err]     at PriorityQueue.percolateDown (internal/priority_queue.js:70:7)
      [err]     at listOnTimeout (internal/timers.js:512:24)
      [err]     at processTimers (internal/timers.js:492:7)
      [err]
      AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

      1 !== 0

          at runTests (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-inspector-break-when-eval.js:65:10)
          at processTicksAndRejections (internal/process/task_queues.js:97:5) {
        generatedMessage: true,
        code: 'ERR_ASSERTION',
        actual: 1,
        expected: 0,
        operator: 'strictEqual'
      }
      1
    ...

------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL        https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type       JS_TEST_FAILURE
Built On   test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
  not ok 2802 sequential/test-perf-hooks
    ---
    duration_ms: 0.788
    severity: fail
    exitcode: 1
    stack: |-
      {
        name: 'node',
        entryType: 'node',
        startTime: 0,
        duration: { around: 426.24277687072754 },
        nodeStart: { around: 0 },
        v8Start: { around: 0 },
        bootstrapComplete: { around: 421.70141792297363, delay: 2500 },
        environment: { around: 0 },
        loopStart: -1,
        loopExit: -1
      }
      assert.js:384
          throw err;
          ^

      AssertionError [ERR_ASSERTION]: v8Start: 250.21231079101562 >= 250
          at checkNodeTiming (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-perf-hooks.js:67:7)
          at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-perf-hooks.js:79:1)
          at Module._compile (internal/modules/cjs/loader.js:1202:30)
          at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
          at Module.load (internal/modules/cjs/loader.js:1051:32)
          at Function.Module._load (internal/modules/cjs/loader.js:947:14)
          at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
          at internal/main/run_main_module.js:17:47 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: false,
        expected: true,
        operator: '=='
      }
    ...

------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL        https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type       JS_TEST_FAILURE
Built On   test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
  not ok 2821 sequential/test-worker-prof
    ---
    duration_ms: 31.410
    severity: fail
    exitcode: 1
    stack: |-
      assert.js:102
        throw new AssertionError(obj);
        ^

      AssertionError [ERR_ASSERTION]: child exited with signal: {
        error: Error: spawnSync /home/iojs/build/workspace/node-test-commit-linux-containered/out/Release/node ETIMEDOUT
            at Object.spawnSync (internal/child_process.js:1070:20)
            at spawnSync (child_process.js:597:24)
            at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-worker-prof.js:51:23)
            at Module._compile (internal/modules/cjs/loader.js:1202:30)
            at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
            at Module.load (internal/modules/cjs/loader.js:1051:32)
            at Function.Module._load (internal/modules/cjs/loader.js:947:14)
            at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
            at internal/main/run_main_module.js:17:47 {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'spawnSync /home/iojs/build/workspace/node-test-commit-linux-containered/out/Release/node',
          path: '/home/iojs/build/workspace/node-test-commit-linux-containered/out/Release/node',
          spawnargs: [
            '--prof',
            '/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-worker-prof.js',
            'child'
          ]
        },
        status: null,
        signal: 'SIGTERM',
        output: [ null, '', '' ],
        pid: 4629,
        stdout: '',
        stderr: ''
      }
          at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-worker-prof.js:57:10)
          at Module._compile (internal/modules/cjs/loader.js:1202:30)
          at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
          at Module.load (internal/modules/cjs/loader.js:1051:32)
          at Function.Module._load (internal/modules/cjs/loader.js:947:14)
          at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
          at internal/main/run_main_module.js:17:47 {
        generatedMessage: false,
        code: 'ERR_ASSERTION',
        actual: 'SIGTERM',
        expected: null,
        operator: 'strictEqual'
      }
    ...

------------------------------ centos7-arm64-gcc6 ------------------------------
URL        https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc6/29838/console
Type       JS_TEST_FAILURE
Built On   test-packetnet-centos7-arm64-2
Reason
  not ok 2866 sequential/test-inspector-break-when-eval
    ---
    duration_ms: 15.633
    severity: fail
    exitcode: 1
    stack: |-
      [test] Connecting to a child Node process
      [test] Testing /json/list
      [err] Debugger listening on ws://127.0.0.1:41871/5e35f89c-2fda-4abb-ac8f-9234ebdca44c
      [err] For help, see: https://nodejs.org/en/docs/inspector
      [err]
      [test] Setting up a debugger
      [err] Debugger attached.
      [err]
      [test] Breaking in the code
      [err] child process crashed, signal SIGSEGV
      Timed out waiting for matching notification (break on file:///home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/fixtures/inspector-global-function.js:9))
      1
    ...

---------------------------------- win-vs2017 ----------------------------------
URL        https://ci.nodejs.org/job/node-compile-windows/nodes=win-vs2017/32291/console
Type       BUILD_FAILURE
Built On   test-rackspace-win2012r2_vs2017-x64-4
Reason
  error C2139: 'v8::internal::SmallOrderedHashSet': an undefined class is not allowed as an argument to compiler intrinsic type trait '__is_convertible_to' [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_initializers.vcxproj]
    c:\workspace\node-compile-windows\node\out\release\obj\global_intermediate\torque-output-root\torque-generated\class-definitions-tq.h(151): note: see declaration of 'v8::internal::SmallOrderedHashSet'
    c:\workspace\node-compile-windows\node\deps\v8\src\codegen\tnode.h(250): note: see reference to class template instantiation 'std::is_convertible<T,v8::internal::Object>' being compiled
            with
            [
-------------------------- cross-compiler-armv7-gcc-6 --------------------------
URL        https://ci.nodejs.org/job/node-cross-compile/nodes=cross-compiler-armv7-gcc-6/28127/console
Type       BUILD_FAILURE
Built On   test-joyent-ubuntu1604_arm_cross-x64-1
Reason
  error: unable to delete 'jenkins-node-test-commit-arm-fanned-1ab2b73b513cc7aa3908c0bd10f1c8238607a4e4-binary-pi1p/cc-armv7': remote ref does not exist
  error: failed to push some refs to 'binary_tmp@169.60.150.88:binary_tmp.git'
  + true
  + JOBS=2
  + CONFIG_FLAGS=' --dest-cpu=arm'
  + make build-ci -j2
--------------------------------- Other builds ---------------------------------

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the ncu-ci command and its existing summary output, then trace how build and test failures are collected across platforms. The work is complete when the output includes grouped build failures and test failures with each test's platform count, matching the example format.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
ci-cd, cli
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.