[Question/Discussion] The Architectural Intent Behind "Dynamo on Ray" and Future Support for Dynamo's Native Autoscaler

Hi NeMo Curator team 👋,

I’ve been exploring the architecture of nemo_curator recently, specifically looking into the integration of NVIDIA Dynamo within the Ray ecosystem (nemo_curator.core.serve.dynamo.infra). I have a few observations and questions regarding the current design and the future roadmap for this module.

1. Current Understanding of the Integration
Looking at the source code, the current "Dynamo on Ray" implementation appears to be a relatively thin wrapper. It leverages Ray's actor pool/subprocess management to statically provision Dynamo workers. However, it seems to completely disable or bypass Dynamo's native, SLA-driven autoscaler (the Planner).

2. Questioning the Current "Thin Wrapper" Approach
This leads me to wonder about the practical value of this specific integration in its current state. NVIDIA Dynamo's core strength lies in its single-hop, extreme-low-latency control plane that dynamically scales Prefill/Decode pools based on real-time FPM (Forward Pass Metrics) like TTFT. By treating Dynamo merely as a static worker managed by Ray, aren't we sacrificing the very autoscaling capabilities that make Dynamo so powerful?

3. Is "Dynamo on Ray" Fundamentally an Anti-Pattern?
When comparing Dynamo's native autoscaler with Ray Core's autoscaler, they essentially aim to achieve the exact same orchestration goal.
Given the architectural clash between Ray's multi-hop, queue-driven control plane and Dynamo's single-hop, SLA-driven planner, is layering them together inherently an anti-pattern? Stacking two control mechanisms trying to solve the exact same problem seems bound to cause conflicts, which might logically explain why the current integration opts for static provisioning.

4. Roadmap and Future Support
Are there any plans in the community to fully support Dynamo's native autoscaling capabilities within Curator? If so, what would the architectural approach be? Does supporting it mean we have to reimplement Dynamo's Planner logic natively on top of Ray Core's autoscaling primitives? Or is there a way to let Dynamo's control plane operate autonomously (treating it as an independent microservice) while still being macro-orchestrated by Ray?

I would really appreciate any insights into the design philosophy here and how the team views the boundary between Ray's orchestration and Dynamo's serving optimizations.

Thanks!

Start by reading the nemo_curator.core.serve.dynamo.infra integration and compare its Ray-managed worker behavior with the Dynamo autoscaler described in the issue. Review the existing discussion and project roadmap, if referenced by maintainers. Done would require an agreed architectural direction and documented support plan, not a localized code change.