`runner_per_model` is tricky to tune: higher than 4 causes hangs on 96-core 8-GPU explorer machine

We observed that Ray by default creates number of processes equal to the number of CPUs on the machine.

Then for explorer: we get 8 models per machine (when a model fits one GPU). So when we set `runner_per_model` to 8 (to have a bit more than we needed), we got frequent hangs (including timeout crashes because of hangs), and often Ray cannot schedule a task and puts status PENDING ALLOCATION.

Also, for some reason we get 80% CPU load on explorer machines, even though there is no real CPU load there, just Ray communication. This does not make sense, as the actual heavy load is GPU...

Why is that? Does Ray makes the actors spin-wait and still consume a lot of actual CPU cycles? (that would be opposite of OS threads which yield when waiting on some I/O)

Would you have advices on setting `runner_per_model` and configuring Ray (should we increase somehow the upper bound on "number of CPUs" in Ray?), so that explorers do not hang? For now, we had to set `runner_per_model=2`, which can cause somehow that not all prompts in the batch can start at the same time (in sync mode), just because of some intermediate actors are busy... Ideally, the only bottleneck should be the vllm, and certainly not any intermediate Ray actors...

We also often observe that Trinity hangs for several hours before explorers/training starts churning, and don't really understand what's happening during this time: what is hanging, what is waiting. For this, desperately per-component log-files are needed with logging before/after any RPC calls...