kubeflow / kubeflow/trainer

Support GRPO-style post-training in Kubeflow Trainer via TrainingRuntime and targeted Torch plugin support

Open
#3,508 8 comments 6 reactions 1 assignee Claimed by @rehaan-patil View on GitHub
area/rl kind/feature
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What you would like to be added?

I would like to propose initial support for GRPO-style post-training in Kubeflow Trainer.
Concretely, a good first step seems to be:

1. add one or more GRPO-focused ClusterTrainingRuntime examples/manifests
2. support a GRPO trainer image + launcher contract
3. extend the Torch plugin only where GRPO needs targeted validation or command mutation
4. keep rollout backend configuration explicit through runtime/config inputs###

### Why is this needed?

Kubeflow Trainer v2 already has a strong direction for distributed training, LLM fine-tuning, and runtime-centric extensibility.

At the same time, LLM post-training is moving beyond SFT-only workflows. Techniques like GRPO are becoming relevant for:

- RLHF-style post-training
- reward-driven alignment
- iterative improvement of open-weight models
- experimentation with smaller or more specialized models after supervised fine-tuning

This seems like a natural area for Trainer to support next because:

- Trainer already supports specialized LLM fine-tuning behavior through runtimes and targeted plugin logic
- the LLM Trainer design already leaves room for future fine-tuning techniques such as RLHF
- there does not appear to be a clear upstream path yet for GRPO specifically
- I searched existing issues for GRPO and did not find a dedicated issue already tracking this.

If there is alignment on the direction, I would be happy to follow up with:

- a narrow implementation PR, or
- a proposal/KEP first, if that is the preferred path

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.