nodejs / nodejs/node-core-utils

Reliablility report is inaccurate

Open
#647 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
314
Forks
137
Avg merge
2d 8h
Merged PRs (30d)
24

Description

I started investigating https://github.com/nodejs/node/issues/44805
when taking a look at https://github.com/nodejs/reliability/issues/388:

Reason sequential/test-watch-mode-inspect
Type JS_TEST_FAILURE
Failed PR 9 (https://github.com/nodejs/node/pull/44787/, https://github.com/nodejs/node/pull/44741/, https://github.com/nodejs/node/pull/44767/, https://github.com/nodejs/node/pull/44770/, https://github.com/nodejs/node/pull/44376/, https://github.com/nodejs/node/pull/44693/, https://github.com/nodejs/node/pull/44686/, https://github.com/nodejs/node/pull/44771/, https://github.com/nodejs/node/pull/44711/)
Appeared test-nearform-macos10.15-x64-2, test-nearform-macos10.15-x64-1
First CI https://ci.nodejs.org/job/node-test-pull-request/46819/
Last CI https://ci.nodejs.org/job/node-test-pull-request/46857/
Example
not ok 3789 sequential/test-watch-mode-inspect
  ---
  duration_ms: 120.67
  severity: fail
  exitcode: -15
  stack: |-
    timeout
    TAP version 13
    (node:37761) ExperimentalWarning: The test runner is an experimental feature. This feature could change at any time
    (Use `node --trace-warnings ...` to show where the warning was created)
  ...


the first PR in the Failed PR row has two CI runs prior to this report: 46819, 46808

the first CI run reports a failure for this based on the previous osx1015 run https://ci.nodejs.org/job/node-test-commit-osx/47578/ that did not actually run in this build (due to java.net.ConnectException: Connection timed out) but Jenkins still considers the child of a previous run to be the child of this run.

this obviously happened in other PRS/CI runs as well, wich makes the report links and count very hard to rely on when investigating flaky test issues

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the reliability report for nodejs/reliability#388 derives its Failed PR, First CI, Last CI, and Appeared links from the referenced Jenkins runs 46819, 46808, 47578, and 46857. Compare parent and child build relationships, including the failed connection in run 47578. Done means report links and counts reflect CI runs that actually executed.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
ci-cd, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.