kubeflow / kubeflow/mpi-operator

Mixed worker configuration is not possible

Open
#791 5 comments 0 reactions 1 assignee Claimed by @Sridhar1030 View on GitHub
Dominant language
Go
Stars
535
Forks
238
Avg merge
2d 3h
Merged PRs (30d)
10

Description

# Issue
Running workers with different configuration is not possible, e.g. one with `nvidia.com/gpu` and one with `amd.com/gpu`

```yaml
Worker:
replicas: 1
template:
spec:
containers:
- image: mpioperator/tensorflow-benchmarks:latest
name: tensorflow-benchmarks
resources:
limits:
nvidia.com/gpu: 2
```

# Why is this relevant
Being able to use different GPU models or vendors is a feature users ask for.

# Solution
Maybe supporting a configuration for specific workers would solve it.

```yaml
Worker-1:
template:
spec:
containers:
- image: mpioperator/tensorflow-benchmarks:latest
name: tensorflow-benchmarks
resources:
limits:
nvidia.com/gpu: 2
Worker-2:
template:
spec:
containers:
- image: mpioperator/tensorflow-benchmarks:latest
name: tensorflow-benchmarks
resources:
limits:
amd.com/gpu: 2
```

I am happy to contribute if this feature is wanted.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.