MPI runLauncherAsNode allocates an extra worker when numNodes is 1
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 39
Description
### What happened?
With runLauncherAsNode: true, EnforceMLPolicy uses max(numNodes-1, 1) for the node PodSet count. For numNodes: 1, this results in one launcher plus one worker, so the requested node count becomes 2. Should numNodes: 1 produce a launcher-only topology, or should it be rejected at admission with a minimum of 2?
### What did you expect to happen?
When numNodes: 1 and runLauncherAsNode: true, the launcher should count as the single node, so the worker count should be 0 - not forced to be a minimum of 1
### Environment
Kubernetes version:
```bash
$ kubectl version
```
Kubeflow Trainer version:
```bash
$ kubectl get pods -n kubeflow-system -l app.kubernetes.io/name=kubeflow-trainer -o jsonpath="{.items[*].spec.containers[*].image}"
```
Kubeflow Python SDK version:
```bash
$ pip show kubeflow
```
### Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
Contributor guide
Research direction
Start at EnforceMLPolicy, focusing on the runLauncherAsNode handling that calculates the node PodSet count. Determine the intended admission behavior for numNodes: 1, then verify that the resulting topology has one launcher and zero workers without increasing the requested node count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 57/100