[RFC] Heterogeneous GPU types for training and generation

This features requires input from the community on if this is needed and what their setup looks like.

This issue tracks the broad feature of the scenario where training happens on higher end GPUs like 100-series and the generation happens on lower spec'd GPUs like L40s. The interconnect between these two GPU types will probably not be fast, so that is a consideration that needs some design; for instance, maybe this necessitates a proxy node during async RL where we have a checkpoint proxy that acts similar to a CDN like prime-rl

https://www.primeintellect.ai/blog/intellect-2-release