kubeflow / kubeflow/mpi-operator

Failing launcher because network is not yet ready

Open
#482 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
535
Forks
238
Avg merge
2d 3h
Merged PRs (30d)
10

Description

Hello,

I created draft implementation of mpijob into [nextflow](https://github.com/nextflow-io/nextflow/pull/3392).

I run into problem that nextflow expects that job does not fail temporary as is the case of launcher job.

Moreover, failing launcher can be for longer time as it might happend that there are not free resources for all the workes (like few days) and launcher will fail as it exhausts all retry attempts.

Can this be fixed that the mpioperator actually spawns workers first and only if all workers are running, it spawns the launcher? Is there any reason for not doing this?

Contributor guide

Open the contributing guide

Research direction

Review the MPIJob launcher and worker scheduling behavior described in this issue, using the draft Nextflow implementation in PR #3392 for context. Determine how launcher failures and retry exhaustion are currently handled; done means workers are available before the launcher starts and temporary resource shortages do not prematurely fail the job.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.