nodejs / nodejs/node

Various flaky tests when running on test-osuosl-aix72-ppc64_be-3

Open
#61,441 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

aix
Dominant language
JavaScript
Stars
122k
Forks
37.3k
Avg merge
4d 2h
Merged PRs (30d)
283

Description

There are a few tests which are showing high levels of inconsistency when run on the test-osuosl-aix72-ppc64_be-3 machine. While there are occasional (<1%) failures showing on some of the tests elsewhere they are showing enough problems on this machine that it should be investigated as it is causing problems showing up on the node reliability reports (and also from some of my own job runs!)

The ones I've noticed - which I'm currently running additional stress tests on are as follows, although this may not be an exhaustive list:

  • parallel/test-repl-preview (Low failure rate - ~10% maybe)
  • node-api/test_threadsafe_function (~90% failure rate on osuosl_be-3)
  • async-hooks/test-emit-after-on-destroyed (~9% failure rate on osuosl_be-3)

1000 iterations of these on OSUOSL machines be-3 and be-4 are occurring in https://ci.nodejs.org/job/node-stress-single-test/638/console. (Note: I have modified that job to allow me to select the OSUOSL AIX machines specifically instead of just by the generic aix72-ppc64 label to allow explicit comparisons. I've also noted that the async-hooks test did fail once in a 100-iteration run on osuosl-be-4 (AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: / 'SIGILL' !== null) and with a similar frequency on that machine on a 1000-iteration run.

Noting that the intermittent stringbytes-external-max was marked flaky last month and it potentially also primarily a problem on this machine (to be confirmed)

I had also raised https://github.com/nodejs/node/issues/61439 on the threadsafe failure but I will close that as this seems to be primarily an issue with the machine setup/configuration as opposed to a general problem with that test.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the 1000-iteration node-stress-single-test job 638 and compare test-osuosl-aix72-ppc64_be-3 with be-4. Reproduce parallel/test-repl-preview, node-api/test_threadsafe_function, and async-hooks/test-emit-after-on-destroyed, then inspect the reported failures and machine differences. Done means identifying whether the AIX node configuration causes the flakiness and documenting or addressing the resulting fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
operating-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.