KEP-3562: Store trial history natively in `OptimizationJob` Status
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
### What you would like to be added?
**[Context](https://github.com/kubeflow/trainer/pull/3565#discussion_r3582000633):**
In the Phase 1 design of [KEP-3562](https://github.com/kubeflow/trainer/pull/3565), the `OptimizationJob` controller reconstructs trial history by querying existing `TrainJobs` in the cluster.
**Problem:**
Relying entirely on `TrainJobs` to reconstruct optimization state is brittle because they are execution resources with independent lifecycles and may be garbage collected. This makes the `OptimizationJob` less self-contained.
**Future Goals (Phase 2):**
* Update the `OptimizationJobStatus` to aggregate and own the optimization state directly.
* Store the list of all trials (strongly-typed hyperparameters + objective scores) natively in the status.
* Decouple the core optimization logic from `TrainJob` persistence.
*Relates to KEP-3562 Master Tracking [Issue](https://github.com/kubeflow/trainer/issues/3562).*
### Why is this needed?
If Kubernetes garbage-collects old `TrainJobs` to save cluster space, the controller currently loses the experiment's history. Storing the history natively inside the `OptimizationJob` ensures the optimization state is never lost.
### Love this feature?
Give it a 👍 We prioritize the features with most 👍
Contributor guide
Assessment
This issue has not been assessed yet.