Unable to have static-graph + isolated build + caching + // builds, working together

Hey, this is a followup of #7110

So, I'm trying to create a build server that would be responsible to maintain the compilation of a "solution" or a group of C# projects as efficiently as possible by coupling:
- static graph
- isolated builds
- caching of results
- parallelized builds

I have also tried to play around `ProjectCachePlugin` to cover some of these aspects, but I'm hitting a restriction in the design of the caching that I'm not sure how to solve.

The main problem I have is `parallelized builds` in conjunction with the others.

Let's take an example. I have this project dependencies:

- ProjRoot
- LibA
- LibLeaf
- LibB
- LibLeaf
- LibC
- LibLeaf

I would expect to issue a build + caching like this:

- Build LibLeaf => store cache results `LibLeaf.cache`
- Build LibA / LibB / LibC concurrently in isolate builds, with `LibLeaf.cache` input. Each build would produce `LibA.cache`, `LibB.cache`, `LibC.cache`
- Build ProjRoot in isolate builds, don't cache the output, but use the cache from `LibA.cache`, `LibB.cache`, `LibC.cache`

The server would handle the caching state, would handle the life-sync with source on the disks (e.g like [up-to-date-check](https://github.com/dotnet/project-system/blob/main/docs/up-to-date-check.md) of VS) with the ultimate benefits that builds could be much faster than even VS today because all the results are cached, so changing one project would not require to recompute the results of project dependencies...

From some [early results](https://twitter.com/xoofx/status/1466082683889586177) from the prototype I did with the existing isolate caching, it can speed up the build on a single csproj by e.g x10 times faster. It's a lot. Extend that to an entire graph and it could be a game changer.

But I have hit the limitation that I initially didn't caught in the [static-graph](https://github.com/dotnet/msbuild/blob/main/documentation/specs/static-graph.md) doc which is that isolate + cache can only happen in a `BuildManager` and the input and output cache is only setup-able per `BeginBuild`/`EndBuild`

It means that I can calculate all the above, only sequentially and single threaded, which is super limited.

So, instead, I have been trying to schedule the graph myself, by handling the scheduling similar to static-graph (so I copied the code [here](https://github.com/xoofx/BenchBuild/blob/59464264d09939c9a7a50d0d378524dbc65ce746/BenchBuild/Builder.cs#L192-L294)), I have added a way to serialize the results to disk ([here](https://github.com/xoofx/BenchBuild/blob/59464264d09939c9a7a50d0d378524dbc65ce746/BenchBuild/Builder.cs#L278)) and thought that I could rely on [project-cache](https://github.com/dotnet/msbuild/blob/main/documentation/specs/project-cache.md) through `ProjectCachePlugin` to load these serialized results.

Unfortunately, I discovered that `ProjectCachePlugin` are also only supported in the `BuildManager` scenario, which makes them useless in a parallel build.

I would have hoped that I could have issued builds by attaching the input/output to a request (instead of Begin/EndBuild), and that it could execute on msbuild Nodes instead.

```c#
buildManager.BeginBuild();

// Iterate on projects per group in // (as it is done in static-graph scheduling
// ....
loop on batch-able groups {
loop on project on group {
var request = new BuildRequestData(...);
request.Inputs = ...;
request.OuputCache = ....;
buildManager.Execute(submission);
var submission = buildManager.PendBuildRequest(request);
submission.ScheduleAsync(...);
}
}
buildManager.EndBuild();
```

The only solution I see is to build my own kind of build server nodes to do that, by hosting a `BuildManager` and performing my own input/output in these nodes... but ouch, that's a bit more work than expected...

Side notes: the current input/output caches load is super limited in a server by only providing file path loading, while I could also maintain a memory cache that could speed things further.

Thoughts?

cc: @rainersigwald