Let's talk about the CI situation

As of recently, especially with https://github.com/nodejs/build/issues/3887, dealing with the CI when preparing a release or when trying to land a PR has become very frustrating. I tried to list all the pain points, let me know if I forgot something:

• The CI is so flaky it's systematically takes several attempts to get a passing one.
• Most folks do not bother to check the failures.
• Even fewer folks are reporting flaky tests (I plead guilty).
• Even fewer folks are investigating those flakes (and I get it, the very nature of flaky tests makes them very hard to reproduce, let alone debugging them).
• Opening a PR to mark test(s) as flaky often receives objections that the flakiness should be solved instead of ignored (which I totally get, those flakes are most of the time due to a bug in node; but it also makes the situation more frustrating for contributing, and also for other projects that build their own node and expect tests not marked as flaky to not be flaky).

I think we need to discuss:

1. Is there a way to better detect flaky tests before landing them?
2. How can we handle the immediate situation?

I'm opening this in the TSC repo, because it's kind of a meta discussion that I'm guessing is not going to be of much interest to folks following nodejs/node, but of course anyone is welcome to participate to the discussion.

No source files or tests are named. Start by reading this discussion and the referenced nodejs/build issue 3887, then review the comment thread for an agreed direction on detecting and handling flaky CI tests. Done would require a concrete, accepted plan rather than a specific code change.