kubeflow / kubeflow/trainer

KEP-3562: [OptimizationJob] Implement and refactor Optuna suggestion service backend

Open
#3,797 3 comments 0 reactions 1 assignee Claimed by @saivedant169 View on GitHub
area/hpo kind/feature kind/plan-kep
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What you would like to be added?

Context:
In [KEP-3562](https://github.com/kubeflow/trainer/pull/3565), Optuna is selected as the primary backend engine for Phase 1 hyperparameter search algorithms (e.g., Random, Grid).

Problem:
The Optuna integration needs to run as an ephemeral sidecar/container service managed directly by the OptimizationJob controller without relying on external databases or stateful storage.

Future Goals:

- Refactor the Optuna wrapper in `kubeflow/trainer` to act as an isolated gRPC service container.

- Ensure in-memory trial tracking within the container during the job's lifecycle.

Relates to KEP-3562 Master Tracking [Issue](https://github.com/kubeflow/trainer/issues/3562).

### Why is this needed?

A clean, self-contained Optuna service ensures low latency during suggestion requests and guarantees clean teardown when the `OptimizationJob` reaches a terminal state (Complete/Failed).

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.