kubeflow / kubeflow/trainer

KEP-3562: Store trial history natively in `OptimizationJob` Status

Open
#3,744 3 comments 0 reactions 1 assignee Claimed by @Sridhar1030 View on GitHub
area/hpo kind/feature kind/plan-kep
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What you would like to be added?

**[Context](https://github.com/kubeflow/trainer/pull/3565#discussion_r3582000633):**
In the Phase 1 design of [KEP-3562](https://github.com/kubeflow/trainer/pull/3565), the `OptimizationJob` controller reconstructs trial history by querying existing `TrainJobs` in the cluster.

**Problem:**
Relying entirely on `TrainJobs` to reconstruct optimization state is brittle because they are execution resources with independent lifecycles and may be garbage collected. This makes the `OptimizationJob` less self-contained.

**Future Goals (Phase 2):**
* Update the `OptimizationJobStatus` to aggregate and own the optimization state directly.
* Store the list of all trials (strongly-typed hyperparameters + objective scores) natively in the status.
* Decouple the core optimization logic from `TrainJob` persistence.

*Relates to KEP-3562 Master Tracking [Issue](https://github.com/kubeflow/trainer/issues/3562).*

### Why is this needed?

If Kubernetes garbage-collects old `TrainJobs` to save cluster space, the controller currently loses the experiment's history. Storing the history natively inside the `OptimizationJob` ensures the optimization state is never lost.

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.