kubeflow / kubeflow/trainer

MPI runLauncherAsNode allocates an extra worker when numNodes is 1

Open
#3,954 4 comments 0 reactions 0 assignees View on GitHub
kind/bug needs-triage
Dominant language
Go
Stars
2.2k
Forks
1.1k
Avg merge
3d 22h
Merged PRs (30d)
39

Description

### What happened?

With runLauncherAsNode: true, EnforceMLPolicy uses max(numNodes-1, 1) for the node PodSet count. For numNodes: 1, this results in one launcher plus one worker, so the requested node count becomes 2. Should numNodes: 1 produce a launcher-only topology, or should it be rejected at admission with a minimum of 2?

### What did you expect to happen?

When numNodes: 1 and runLauncherAsNode: true, the launcher should count as the single node, so the worker count should be 0 - not forced to be a minimum of 1

### Environment

Kubernetes version:
```bash
$ kubectl version

```
Kubeflow Trainer version:
```bash
$ kubectl get pods -n kubeflow-system -l app.kubernetes.io/name=kubeflow-trainer -o jsonpath="{.items[*].spec.containers[*].image}"

```
Kubeflow Python SDK version:
```bash
$ pip show kubeflow

```

### Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Contributor guide

Open the contributing guide

Research direction

Start at EnforceMLPolicy, focusing on the runLauncherAsNode handling that calculates the node PodSet count. Determine the intended admission behavior for numNodes: 1, then verify that the resulting topology has one launcher and zero workers without increasing the requested node count.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
57/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.