ai-dynamo / ai-dynamo/dynamo

Contribution offer: Terraform (IaC) deploy example - Dynamo on Nebius managed K8s

Open
#12,901 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

deployment::k8s docs operator
Dominant language
Rust
Stars
8.1k
Forks
1.6k
Avg merge
2d 22h
Merged PRs (30d)
446

Description

Hi team, following a conversation with an NVIDIA contact who suggested opening an issue here, I'd like to offer a cloud + Terraform deployment example for Dynamo.

Repo: https://github.com/CloudThrill/nvidia-dynamo-stack-terraform (Nebius MK8s today, structured for per-cloud subfolders)

How it differs from the existing k8s examples. The current examples start from "you have a cluster with the operator installed" and walk through manifests. This one starts from an empty cloud project: terraform apply provisions the VPC, managed Kubernetes, GPU node group, ingress + cert-manager, kube-prometheus-stack, the dynamo-platform chart and the DGD, one command rather than a copy-paste sequence. As far as I can tell there's no IaC example in the repo today.

Dynamo on Nebius MK8s deployment overview

What's in it: all verified on a live 1.3.0 cluster:

  • Multi-model behind one shared Frontend on a single GPU, including the per-worker --endpoint dyn://$DYN_NAMESPACE-$DYN_NAMESPACE_WORKER_SUFFIX.<name>.generate needed so the second model isn't rejected on the default backend component
  • Operator-managed ingress + Let's Encrypt TLS (the operator references a TLS secret it doesn't create, the example ships the cert-manager Certificate that fills it)
  • Prometheus/Grafana with the shipped Dynamo + operator dashboards, including the monitor selector relaxation needed for them to be scraped
  • Optional KV-aware routing example (router-mode=kv + worker KV events): verified: 20/20 shared-prefix requests routed to the cache-warm replica
  • Measured deployment timings and a troubleshooting section built from real failures
  • Real HTTPS URL endpoints no IP no port forwarding during the demos

Pinned to Dynamo 1.3.0 / vllm-runtime:1.3.0.

Placement : your call entirely; you know the repo layout better than I do. My instinct would be a self-contained terraform/nebius/ entry alongside the existing k8s examples: same shape as what's already there, just with the cluster provisioning included, so it stays isolated from the rest of the tree.

If carrying IaC in-tree isn't something the project wants, a short docs page linking out works too, though it'd be easier to find sitting with the other examples.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the linked CloudThrill/nvidia-dynamo-stack-terraform repository and the existing Kubernetes examples in this repository. Confirm whether the contribution should be an in-tree terraform/nebius/ example or a documentation link, then verify that the deployment, monitoring, TLS, and troubleshooting flow is reproducible against the pinned Dynamo 1.3.0 setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, prometheus, terraform
Domain
cloud, devops, infrastructure, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.