kubeflow / kubeflow/mpi-operator

Keep Launcher Alive to Retrieve Logs on Error

Open
#354 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
535
Forks
238
Avg merge
2d 3h
Merged PRs (30d)
10

Description

Hi I'm a relatively new user. I'm using my own dockerfile for a project that uses MPICH, and I have been able to successfully deploy the project. However, sometimes I'll be testing features changes and then I'll get an error which terminates the launcher.

I understand that this project is designed to automatically restart the launcher `backoffLimit` times, but even when I set the `backoffLimit` to 1, the launcher terminates so I am unable to retrieve the logs. What is the best way to retrieve the logs in this situation? Is there an option to avoid terminating the launcher once the number of failures have reached the backoff limit?

Contributor guide

Open the contributing guide

Research direction

No file, test, or entry point is named. Start by tracing how launcher failures and backoffLimit are handled, then determine whether retaining the launcher or documenting another log-retrieval path is intended; done means logs remain retrievable after the failure limit or the supported alternative is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
dockerfile, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.