Observation is not available for early stopping Trials with retain: false
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 542
- PR merge metrics
- No merged PRs in 30d
Description
/kind bug
If Trial is early stopped and `retain` parameter is equal to `false`, Trial observation is not available.
Currently, early stopping condition is [complete condition](https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/trials/v1beta1/util.go#L96) for the Trial. We update Trial condition in [early stopping service](https://github.com/kubeflow/katib/blob/master/pkg/earlystopping/v1beta1/medianstop/service.py#L137).
When Trial is complete, we [delete Trial's job](https://github.com/kubeflow/katib/blob/master/pkg/controller.v1beta1/trial/trial_controller.go#L308-L317), which means, we don't [update Trial observation after](https://github.com/kubeflow/katib/blob/master/pkg/controller.v1beta1/trial/trial_controller.go#L227).
For some cases, marking Trial complete when it is early stopped is not correct. For example, when Trial is a Tekton Pipeline and training job is not the latest step of the Pipeline. I am not sure if it is correct to mark Trial complete, after Trial is early stopped.
Maybe we should mark Trial complete only when Trial's job is succeeded and observation is updated.
We should investigate how we can deal with this situation.
/cc @gaocegege @johnugeorge
Contributor guide
Research direction
Read pkg/apis/controller/trials/v1beta1/util.go, the early stopping service at pkg/earlystopping/v1beta1/medianstop/service.py, and the trial controller paths around lines 227 and 308. Trace how early stopping, job deletion, and observation updates interact when retain is false. Done means the lifecycle is defined and an early-stopped Trial does not lose its observation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, python
- Domain
- backend, distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100