kubeflow / kubeflow/katib

Observation is not available for early stopping Trials with retain: false

Open
#1,377 2 comments 0 reactions 0 assignees View on GitHub
kind/bug lifecycle/frozen
Dominant language
Python
Stars
1.7k
Forks
542
PR merge metrics
No merged PRs in 30d

Description

/kind bug

If Trial is early stopped and `retain` parameter is equal to `false`, Trial observation is not available.

Currently, early stopping condition is [complete condition](https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/trials/v1beta1/util.go#L96) for the Trial. We update Trial condition in [early stopping service](https://github.com/kubeflow/katib/blob/master/pkg/earlystopping/v1beta1/medianstop/service.py#L137).
When Trial is complete, we [delete Trial's job](https://github.com/kubeflow/katib/blob/master/pkg/controller.v1beta1/trial/trial_controller.go#L308-L317), which means, we don't [update Trial observation after](https://github.com/kubeflow/katib/blob/master/pkg/controller.v1beta1/trial/trial_controller.go#L227).

For some cases, marking Trial complete when it is early stopped is not correct. For example, when Trial is a Tekton Pipeline and training job is not the latest step of the Pipeline. I am not sure if it is correct to mark Trial complete, after Trial is early stopped.

Maybe we should mark Trial complete only when Trial's job is succeeded and observation is updated.
We should investigate how we can deal with this situation.

/cc @gaocegege @johnugeorge

Contributor guide

Open the contributing guide

Research direction

Read pkg/apis/controller/trials/v1beta1/util.go, the early stopping service at pkg/earlystopping/v1beta1/medianstop/service.py, and the trial controller paths around lines 227 and 308. Trace how early stopping, job deletion, and observation updates interact when retain is false. Done means the lifecycle is defined and an early-stopped Trial does not lose its observation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, python
Domain
backend, distributed-systems, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.