llvm-lit feature to run resource sensitive tests individually, after the other parallel tests

In some cases we have a test that has to rely in real time. It cannot use a synchronisation mechanism like a mutex, it has to use some real time thing. Usually because it is testing a timeout feature, or a time reporting feature.

The problem with these tests is that they can be randomly flakey if the OS scheduler decides to stall a part of the test. For instance in https://github.com/llvm/llvm-project/issues/217733 the cause was that the client was supposed to wait a certain time, then timeout. The test server did this by sleeping more than that time, then returning a succesful error code in case the client had not timed out correctly.

If the client is stalled by scheduling for more than the timeout, or at timeout-0.0001s to timeout+0.0001s it looks like the test has failed, but it wasn't the code's fault. With that test structure there is nothing we can do about that but increase timeouts and hope. Which is a pretty successful strategy but if you've run a fleet of buildbots you'll know it isn't great, plenty of people still get false failure reports.

Examples of this problem are:
* https://github.com/llvm/llvm-project/issues/217733, described above. Which had to be fixed by narrowing the test case. If we wanted to test that we adhered accurately to the timeout, we would have to risk flakey results again.
* https://github.com/llvm/llvm-project/issues/209874 where we wait for threads to appear in the LLDB TUI. We cannot use any other sync mechanism because that requires the process to stop, and we don't know when it will stop.
* https://github.com/llvm/llvm-project/pull/212574 "With the default 250ms expression timeout, the kernel sometimes doesn't schedule the detached child fast enough, causing waitpid() to still be blocking when the timeout fires and the expression gets interrupted.". This one is not quite an issue with the test itself, but it's the same idea, the scheduler gets in the way.
* https://github.com/llvm/llvm-project/pull/221239 - I'm not sure on this one, but these tests have had sync issues in the past.
* https://github.com/llvm/llvm-project/issues/137660 - TestDAP attach is often timing out, I think waiting for the attachee or attacher.

It's not just lldb, though it often is because we spawn a lot of sub-process during testing.
* https://github.com/llvm/llvm-project/blob/f2d4b9dfc7e5bba16f6b3b46469e6cad619370db/libcxx/test/std/thread/thread.mutex/thread.mutex.requirements/thread.timedmutex.requirements/thread.timedmutex.class/try_lock_until.pass.cpp unlocks a mutex without a try timeout. Tbf, this one is very stable but it does mention race possible race conditions.
* https://github.com/llvm/llvm-zorg/blob/1c70b285e66212494554a269c0a2fd9331133c28/buildbot/osuosl/master/config/builders.py#L368 - I had to disable lit's timeout tests on Linaro's bots because they were too unstable.

There is a feature in lit that can limit the number of a type of test that run in parallel. However this is about resource sharing, for instance there is one simulator, don't run more than one simulator test. It does not mean that that one test in that category cannot be run in parallel with other tests not in that category.

compiler-rt has a `test_retry_attempts` feature that probably would help, but each retry is in theory just as vulnerable to this same problem. libcxx has `ALLOW_RETRIES`.

We could just increase timeouts massively, but at some point your internal test timeouts become the same as that of the whole suite, and if even that doesn't fix the flakes, you've nowhere to go from that point.

So what I'm thinking about is roughly:
* `REQUIRES: exclusive`
* All tests marked this way are run sequentially, without other parts of the suite running.
* This can happen at the start or end of a run, doesn't matter.
* So we have a greater chance that the test has the resources it needs to complete in the time that the author guessed it would (someone could be running other things besides llvm, but this is less likely on the bots which is mostly what I care about).
* We would need to be strict so that the test suite doesn't turn into a single threaded affair. Tests that can use synchronization mechanisms should do so. Only when there is no other way would you reach for this marker.

It seems to make sense to start this in lldb as that's where the most issues are, but because llvm-lit controls the scheduling it will need some changes.

I don't have any code for this at this time, but this can at least be an issue on which to collect tests that might benefit from it.

Start with llvm-lit's scheduling implementation and its existing parallel resource-limit feature, then review the cited LLDB timeout issues and pull requests for motivating cases. Define how an `exclusive` test marker is recognized and scheduled after or before the parallel tests. Done means marked tests run sequentially without other suite tests and ordinary tests remain parallel.

Written by the indexing model from the issue text.