[Sandbox] OpenRL

### Project summary

Self-hosted kubernetes-native post training (RL) API for LLMs

### Project description

OpenRL implements the post-training APIs used to fine-tune language models (supervised fine-tuning and reinforcement learning) as a service you run on your own Kubernetes cluster. The API surface is compatible with Tinker, so researchers write their training loop as plain Python with the Tinker SDK and existing tinker-cookbook recipes run unchanged. Four operations are the whole contract between the researcher and the platform: `forward_backward`, `optim_step`, `sample`, and `save_weights`.

The project exists because agentic RL on LLMs couples two very different jobs. Running a single RL loop today means coordinating datasets, environments, reward signals, training loops, inference mismatches, hardware allocation, and the infrastructure underneath all of it. OpenRL separates those concerns the way Kubernetes separated application development from infrastructure operations: AI researchers own the recipe, data, environment, reward, and loss; platform engineers own capacity, scheduling, weight synchronization, and the service itself. Each side can evolve and scale independently.

Under the API, OpenRL decomposes every RL job into units the cluster can place (trainer steps, sampling batches, weight syncs) and runs them on shared accelerators:

* A **multi-tenant queue** buffers work from many concurrent jobs; trainer and sampler workers pull from it and tenants are served round-robin.
* A **placement controller** (a Kubernetes controller with `Workload` and `ClaimLedger` custom resources) sizes accelerator claims to the work using Kubernetes Dynamic Resource Allocation (DRA), bin-packs units that co-locate well, and lets kube-scheduler and DRA own device inventory. Unmet demand surfaces as a standing pending claim, which is an honest signal for the cluster autoscaler.
* **Accelerator time-slicing** built on the llm-d snapshot agent parks a worker's GPU state in host memory and restores it, so full fine-tuning jobs from different tenants take turns on one GPU. Synchronous on-policy RL otherwise leaves accelerators idle while environments run and rewards are computed; OpenRL fills those gaps with another job's step.
* **Delta weight sync** moves only the parameters that changed from trainer to sampler and patches them into the sampler's VRAM in place, over ordinary shared storage. This decouples trainer and sampler placement and cut the measured hand-off for an 8B model from 57 seconds to under one second on the critical path.
* **LoRA and full fine-tuning** are both supported. LoRA jobs share a base model inside one multi-tenant worker; full fine-tuning jobs get their own process and take turns. Workers are packaged as container images (PyTorch trainer, vLLM sampler today), so new frameworks and accelerators plug in without touching a researcher's loop.

Everything in a deployment is an ordinary Kubernetes object (Deployments, DaemonSets, custom resources, ResourceClaims, pods) that operators observe with kubectl and existing monitoring.

OpenRL deliberately covers only the training and sampling half of post-training and composes with the rest of the ecosystem for everything else. RL environments, sandboxes for agent rollouts, and LLM-as-a-judge reward scoring are not reimplemented inside OpenRL. Researchers run environments in agent-sandbox (a Kubernetes sub-project under SIG Apps) through its Python bindings, and serve judge models or any additional inference through llm-d. Because the RL loop is user code that calls these pieces through their own APIs, teams pick the components that fit their workflow and OpenRL stays small.

OpenRL is Apache 2.0 licensed, started in February 2026, and is currently a research preview. It has been used internally on realistic workloads, including fine-tuning open models against a long-horizon legal agent benchmark, which shaped the reliability priorities on the roadmap.

### Project vs Reference Architecture/Implementation

- [x] This is a reusable open source project designed for broad adoption, not a reference architecture, reference implementation, or demonstration of patterns

### Org repo URL (provide if all repos under the org are in scope of the application)

N/A

### Project repo URL in scope of application

https://github.com/gke-labs/open-rl

### Additional repos in scope of the application

_No response_

### Website URL

https://github.com/gke-labs/open-rl

### Roadmap

https://github.com/gke-labs/open-rl/blob/main/ROADMAP.md

### Roadmap context

The roadmap covers FY 2026 and is organized as focus areas with initiatives under each, in rough priority order. The areas are:

* **Reliability and production readiness.** Loud failures in the reward and grading path, reliable queue semantics for training steps, checkpoint and resume guarantees including optimizer state, resiliency to preemptible workers, secrets handling through Kubernetes primitives, per-tenant quotas and fairness, and a versioned API with an upgrade path. This is first because on real workloads most lost time went to the environment rather than the model.
* **Multi-GPU and advanced topologies.** Extend the scheduler and Workload spec to multi-GPU trainers and samplers within a node (samplers first via vLLM tensor parallelism, then in-node FSDP trainers, then a Megatron engine), plus capacity accounting for LoRA runtimes. Multi-host training is an explicit non-goal for the year.
* **Expand accelerator support.** TPUs first, as a deliberate test that the four-operation API plus a container image per worker is enough of a contract to swap the runtime underneath. AMD and Intel are welcome from partners but not owned by the maintainers this year.
* **Expand supported models.** Dense models to the 30B range, MoE models, multi-model tenancy across base models in one deployment, and a documented path for onboarding new model families.
* **Observability, management dashboard, and CLI.** Read-only first, with a CLI whose output agentic tools can consume, because an increasing share of AI research and system administration is done by agents.
* **Recipes, guides, and docs.** Realistic domain recipes (legal, finance), reproductions of published RL results, widening Tinker API coverage, and deployment guides for GKE and for Kubernetes with DRA in general.
* **Community.** Deployment and contributor guides, CI and presubmit improvements, good-first issues, a public project board mirroring the roadmap, regular office hours, and a regular release cadence.

Direction in one sentence: the researcher's loop should stay the same across models and across accelerators, and everything below the API should be swappable and operable with standard Kubernetes tooling.

### Contributing guide

https://github.com/gke-labs/open-rl/blob/main/CONTRIBUTING.md

### Code of Conduct (CoC)

https://github.com/gke-labs/open-rl/blob/main/CODE_OF_CONDUCT.md

### Adopters

_No response_

### Maintainers file

https://github.com/gke-labs/open-rl/blob/main/MAINTAINERS.md

### Security policy file

https://github.com/gke-labs/open-rl/blob/main/SECURITY.md

### Standard or specification?

OpenRL is not itself a standard, but it implements an existing, publicly documented API: the Tinker post-training API (four primitives for training and sampling plus their supporting request, future, and checkpoint calls). OpenRL is an independent, open-source, self-hostable implementation of that interface. Compatibility is tracked in https://github.com/gke-labs/open-rl/blob/main/docs/tinker-client-compatibility.md. The project also defines two Kubernetes custom resources (`Workload`, `ClaimLedger`) that describe a post-training worker and its accelerator seat; these are project APIs, not a proposed standard.

### Business product or service to project separation

OpenRL is developed by GKE Labs, a team at Google. It is not the upstream of any Google product or service and is not a required dependency of one. Specifically:

* **Not tied to GKE.** OpenRL runs on any Kubernetes 1.34+ cluster with Dynamic Resource Allocation. The repository ships a kind-based developer cluster and GKE is documented as one reference deployment among the supported paths (`k8s/deploy/` includes shared-storage, Lustre, kind, and single-process variants). No GKE-only APIs are required.
* **Not tied to Google models.** Gemma and Qwen are both tested out of the box, and the model onboarding path is model-family neutral. Gemma is a model the project runs, not a dependency.
* **Not tied to a hosted service.** OpenRL is explicitly self-hosted. Google does not sell OpenRL as a service. Internal use has been dogfooding the open-source code on realistic workloads to find infrastructure gaps, and the fixes flow back to the public repository.

All development happens in the public GitHub repository under the project's governance (GOVERNANCE.md), which commits to vendor neutrality: project direction, maintainer selection, and release decisions are made on project merit, not employer affiliation. Today all five maintainers are Google employees; the community section of the roadmap (contributor guides, good-first issues, office hours, a public project board) and the move to a neutral foundation are the plan for changing that.

### Why CNCF?

The industry is evolving from its generative foundations to include agentic approaches, and reinforcement learning is central to that shift. Post-training infrastructure is at the point container orchestration was before Kubernetes: every team is coordinating datasets, environments, rewards, training loops, inference, and hardware by hand, and the tooling couples AI research to infrastructure so tightly that neither side can move on its own. Decoupling those concerns behind an API made application development and operations independently tractable once; OpenRL takes the same approach to RL.

We want to contribute OpenRL to the CNCF now, before the ecosystem fragments into proprietary silos, so that the RL infrastructure stack can build on an open, self-hostable implementation in a neutral home. Frontier model builders, hyperscalers, and enterprises that would avoid a single-vendor project can adopt and contribute to a CNCF project. Vendor neutrality is already written into our governance; the foundation makes it credible.

OpenRL is also cloud native in the literal sense. It extends Kubernetes rather than bypassing it: placement is a Kubernetes controller with custom resources, accelerators are allocated through Dynamic Resource Allocation, workers are OCI images in pods, capacity signals go to the cluster autoscaler, and operators use kubectl and standard metrics. It does this without introducing a separate distributed runtime such as Ray for the platform tasks in RL. And it composes with, rather than duplicates, the projects around it: agent-sandbox for RL environments, llm-d for judge and inference serving, Kueue and Volcano for quota and admission, Prometheus and OpenTelemetry for observability. We believe this fills a gap in the Kubernetes story for a complex and growing workload class, and that the CNCF is the right community to review, shape, and elevate it as the Kubernetes-native practice for RL post-training.

There is a further reason the CNCF is the right home. CNCF end users increasingly care about a **sovereign AI stack**: open-weights models, run and adapted on infrastructure they control, without dependence on a single vendor's hosted service. Inference and serving in that stack are being addressed by projects in the ecosystem. Post-training is not. Taking an open-weights model and fine-tuning it to an organization's own workflows, data, and domain is the step that turns a general model into an asset for a specific industry, and today that step is under-served by open, self-hostable infrastructure. OpenRL fills exactly that slot: an open-source post-training service that runs on your own Kubernetes cluster, on your own accelerators, against your own data, behind an API compatible with the tooling researchers already use. A neutral foundation is where that component belongs.

For the project, CNCF membership provides neutral governance and IP ownership, access to the TOC's and TAGs' review, a path for contributors from other organizations to become maintainers, and the visibility of the landscape and CNCF events. We feel the sooner this transfer occurs the better for the project and for the community, and OpenRL's foundation is solid enough to make the move.

### Benefit to the landscape

Post-training and RL for LLMs is a fast-growing workload that the landscape does not yet address as a first-class Kubernetes concern. Existing options are either full RL frameworks that own the loop and bring their own distributed runtime, or hosted services that keep the infrastructure proprietary. OpenRL adds a third thing: a self-hosted API boundary between researchers and platform engineers, implemented with Kubernetes primitives.

Concrete differentiators:

* **An API contract, not a framework.** Users keep full control of their loop and data. The four-operation surface is compatible with an existing, widely used SDK, so recipes and cookbooks carry over.
* **Multi-tenancy on shared accelerators.** Concurrent RL jobs share one pool, with time-slicing for full fine-tuning and shared base models for LoRA. This directly attacks GPU idle time in synchronous on-policy RL, which is the dominant cost in this workload.
* **DRA-native accelerator scheduling.** A reusable pattern for dynamic, heterogeneous accelerator placement (mixed L4 and H100 pools today) that defers to kube-scheduler and DRA for inventory and gives autoscalers honest demand.
* **Delta weight synchronization over ordinary storage.** Decouples trainer and sampler placement and avoids reloading full checkpoints, a technique other projects in the landscape can adopt.
* **Pluggable workers as container images.** Frameworks and accelerators are additive: PyTorch and vLLM on NVIDIA today; TPUs, Megatron, MaxText, and SGLang on the roadmap, none requiring changes to the researcher's code.
* **Composability instead of duplication.** OpenRL does not ship its own sandboxes, environments, judge serving, or quota system. It plugs into agent-sandbox for environments, llm-d for judge and inference serving, and Kueue or Volcano for admission, which strengthens those projects' place in the landscape rather than competing with them.
* **The missing piece of a sovereign AI stack.** The landscape already covers serving open-weights models on your own cluster. OpenRL adds the ability to adapt them there too, so an organization can go from an open checkpoint to a model fine-tuned on its own data without leaving infrastructure it controls.

For platform teams already on Kubernetes, OpenRL lets them offer RL-as-a-Service with the operational model they already have, rather than standing up a second control plane.

### Cloud native 'fit'

OpenRL fits at the intersection of **orchestration and management** and **AI/ML** in the landscape: it is a Kubernetes-native platform layer for a specific workload class, built the way Kubernetes extensions are built.

It embodies cloud native principles as follows:

* **Declarative, API-driven.** Post-training workers are declared as `Workload` custom resources; a controller reconciles them into claims and pods and reports status back on the object. Researchers interact through a versioned HTTP API.
* **Container-packaged, immutable workers.** Trainers and samplers are OCI images. Swapping a framework or accelerator is swapping an image.
* **Dynamic, elastic resource use.** Accelerators are allocated through Kubernetes Dynamic Resource Allocation and shared across tenants; unmet demand becomes a pending claim the cluster autoscaler can act on.
* **Loosely coupled components.** API server, queue, scheduler, trainers, samplers, and the time-slicing daemon are separate services that communicate through the queue, shared storage, and the Kubernetes API.
* **Observable with standard tooling.** kubectl for every object, DCGM-based GPU metrics for Prometheus, OpenTelemetry tracing in the gateway.
* **Multi-tenant by design.** Fairness is a unit of the scheduler (owners get turns, not processes), and per-tenant quotas are on the roadmap.
* **Portable.** The same manifests run on kind and on managed Kubernetes; the same client code runs against a laptop and a cluster.
* **Composable, single-responsibility.** OpenRL owns training and sampling and nothing else. Environments, sandboxes, reward models, quota, and observability come from neighbouring projects through their own APIs, in the same way a cloud native application is assembled from focused services rather than one monolith.
* **Sovereignty-preserving.** Models, data, checkpoints, and accelerators all stay on the operator's cluster. There is no call-home and no dependence on a hosted control plane, which is what makes OpenRL usable as the post-training layer of a sovereign AI stack.

### Cloud native 'integration'

* **Kubernetes** (graduated): OpenRL is a Kubernetes extension. It depends on custom resources, controller-runtime, Dynamic Resource Allocation (GA in 1.34) for accelerator claims, kube-scheduler for placement, DaemonSets for node-local time-slicing, and the cluster autoscaler for capacity.
* **containerd / OCI** (graduated): all workers, the gateway, and the scheduler ship as OCI images.
* **Prometheus** (graduated): GPU utilization and memory are exported through the DCGM exporter for Prometheus scraping; per-step training metrics in a standard exportable format are on the roadmap.
* **OpenTelemetry** (incubating): the gateway and workers emit OpenTelemetry traces for request and training-step lifecycles.
* **Helm / Kustomize** (Helm graduated; Kustomize is a Kubernetes SIG-CLI project): deployments are Kustomize bases and overlays today.
* **Kueue** (Kubernetes SIG project) and **Volcano** (incubating): OpenRL's per-tenant quota and fairness work is intended to integrate with existing batch queueing and quota systems rather than duplicate them; Kueue is the first target.
* **Kubeflow** (incubating): complementary. Kubeflow Trainer runs training jobs; OpenRL provides a serving-style post-training API and multi-tenant accelerator sharing. Running OpenRL workers through Kubeflow Trainer is a natural integration.
* **KServe** (incubating) and the Gateway API Inference Extension (Kubernetes SIG): adjacent on the inference side. OpenRL's sampler is a vLLM worker coupled to training; models produced by OpenRL are served by these projects.
* **llm-d** (CNCF): OpenRL depends on llm-d in two ways. Its snapshot agent provides the GPU checkpoint and restore that OpenRL's time-slicing is built on, and llm-d's distributed serving stack is how users run LLM-as-a-judge reward models and any additional inference their loop needs, so OpenRL does not carry its own serving layer.
* **agent-sandbox** (Kubernetes sub-project under SIG Apps): the sandboxed execution environment for agentic RL rollouts. Researchers build their RL environments on agent-sandbox and drive them from the training loop through its Python bindings; OpenRL never sees the environment and does not need to. This is the composition that keeps environments out of the post-training service.
* **NVIDIA DRA driver for GPUs** (open source): the DRA driver OpenRL's scheduler targets today.

### Cloud native overlap

* **Kubeflow Trainer** (incubating): both run ML training on Kubernetes. Kubeflow Trainer launches and manages training jobs as Kubernetes resources; OpenRL exposes a long-lived, multi-tenant post-training API in which many researchers' loops share workers and accelerators. The overlap is in launching worker pods; OpenRL's scheduler does this today because of the accelerator sharing and time-slicing semantics it needs. We see Kubeflow Trainer as a possible execution backend for OpenRL workers rather than a competitor.
* **Volcano** (incubating) and **Kueue** (Kubernetes SIG): overlap in fairness, quotas, and gang scheduling. OpenRL's scheduler is deliberately narrow: it decides which accelerator claim an RL unit sits on and who takes the next turn, and defers device inventory to DRA and pod placement to kube-scheduler. Quota and admission belong in Kueue or Volcano, and integrating with them is on the roadmap.
* **KServe** (incubating) and **llm-d** (CNCF): overlap only in that OpenRL runs vLLM. OpenRL's sampler exists to produce rollouts for training and to accept in-place weight updates from a co-scheduled trainer, which is not a model-serving concern. General inference, including judge models, is delegated to llm-d rather than duplicated.
* **agent-sandbox** (Kubernetes SIG Apps): no functional overlap. OpenRL intentionally does not provide sandboxes or environments and relies on agent-sandbox for them.

### Similar projects

* **verl** (ByteDance), **OpenRLHF**, **SkyRL** (NovaSky), **prime-rl** (Prime Intellect), **NeMo-RL** (NVIDIA), **TRL** (Hugging Face): RL and post-training frameworks. They own the training loop and typically bring their own distributed runtime (often Ray). OpenRL is not a framework; it is an API service underneath a loop the user owns, and it targets Kubernetes-native scheduling and multi-tenancy rather than single-job throughput. Several of these inspired OpenRL and are acknowledged in the README.
* **Tinker** (Thinking Machines): the hosted post-training API whose interface OpenRL implements. Tinker is a managed service on the vendor's hardware; OpenRL is the self-hosted, open-source implementation for your own cluster.
* **Ray / KubeRay**: a general distributed runtime that many RL frameworks build on. OpenRL intentionally uses Kubernetes primitives (controllers, DRA, DaemonSets) instead of a second runtime.
* **Kubeflow Trainer**, **Volcano**, **Kueue**: see the overlap section above.

### Landscape

No. We will submit a listing under the AI/ML category alongside this application.

### Insights

No, not yet.

### Trademark and accounts

- [x] If the project is accepted, I agree to donate all project trademarks and accounts to the CNCF

### IP policy

- [x] If the project is accepted, I agree the project will follow the CNCF IP Policy

### Will the project require a license exception?

N/A - Project uses Apache 2.0 license.

### Project "Domain Technical Review"

N/A

### Application contact email(s)

smartensson@google.com, sunilarora@google.com, stevenlinde@google.com, jmacleod@google.com

### Contributing or sponsoring entity signatory information

If an organization:
| Name | Address | Type (e.g., Delaware corporation) | Signatory name and title | Email address |
|-----------|-----------|-----------|-----------|-----------|
|Mike Bufano | | | Program Manager • Engineering | bufano@google.com |
|April Kyle Nassi | | | Manager, Open Source Programs Office | anassi@google.com |
|Sven Martensson | | | Program Manager, Cloud Programs | smartensson@google.com |

Or, if an individual or individual(s):
| Name | Country | Email address |
|-----------|-----------|-----------|
| | | |
| | | |
| | | |

### CNCF contacts

Karena Angell, TOC Chair
Allison Price

### Additional information

_No response_