llvm-lit max-failures has a race condition

The --max-failures flag for llvm-lit has a race condition, even with -j1.

If you set max-failures=1, and then have 2 rapid test failures early in the run, both can be reported as failures before the code to count how many tests have failed has run.

As far as I can tell, the tests start running asyncronously as soon as they're added to the pool on line 122 of lit/lit/run.py
But the check for the number of failures doesn't happen until line 156.

It would be expected that there could be a race condition if you were running tests in multiple threads, but with -j1 you would expect there not to be.

I'm unsure what the correct fix is here.
A full removal of the race condition would require a lot more locking which would slow down test execution.
Perhaps having a (duplicated) check in the display code to track the max number of failures would work?
Or have a special case for -j1 which runs everything syncronously.

It's also unclear how vital this is to fix, although if we're going to have a max-failures flag it probably ought to work

Start in lit/lit/run.py around lines 122 and 156, tracing when tests enter the pool and when failure counts are checked. Reproduce the issue with --max-failures=1 and -j1 using two rapid failures, then inspect the existing tests or runner behavior to determine how the failure limit should be enforced. Done means the configured maximum is respected without unnecessarily slowing test execution.

Written by the indexing model from the issue text.