kubeflow / kubeflow/mpi-operator
Failing launcher because network is not yet ready
- Dominant language
- Go
- Stars
- 535
- Forks
- 238
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 10
Description
Hello,
I created draft implementation of mpijob into [nextflow](https://github.com/nextflow-io/nextflow/pull/3392).
I run into problem that nextflow expects that job does not fail temporary as is the case of launcher job.
Moreover, failing launcher can be for longer time as it might happend that there are not free resources for all the workes (like few days) and launcher will fail as it exhausts all retry attempts.
Can this be fixed that the mpioperator actually spawns workers first and only if all workers are running, it spawns the launcher? Is there any reason for not doing this?
Contributor guide
Research direction
Review the MPIJob launcher and worker scheduling behavior described in this issue, using the draft Nextflow implementation in PR #3392 for context. Determine how launcher failures and retry exhaustion are currently handled; done means workers are available before the launcher starts and temporary resource shortages do not prematurely fail the job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100