kubeflow / kubeflow/examples

how to run pytorch mnist ddp

Open
#1,040 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jsonnet
Stars
1.5k
Forks
752
PR merge metrics
No merged PRs in 30d

Description

I have kubeflow deployed now, but there is a problem running the official mnist example, how should I solve it? The yml of PytorchJob is as follows:
```
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
name: pytorch-mnist-ddp-gpu
namespace: kubeflow-user-example-com
spec:
pytorchReplicaSpecs:
Master:
replicas: 1
restartPolicy: OnFailure
template:
spec:
containers:
- image: gcr.io/kubeflow-examples/pytorch-mnist-ddp-gpu
name: pytorch
resources:
limits:
cpu: '1'
memory: 4Gi
nvidia.com/gpu: 1
volumeMounts:
- mountPath: /mnt/kubeflow-gcfs
name: kubeflow-gcfs
volumes:
- name: kubeflow-gcfs
persistentVolumeClaim:
claimName: kubeflow-gcfs
readOnly: false
Worker:
replicas: 2
restartPolicy: OnFailure
template:
spec:
containers:
- image: gcr.io/kubeflow-examples/pytorch-mnist-ddp-gpu
name: pytorch
resources:
limits:
cpu: '1'
memory: 4Gi
nvidia.com/gpu: 1
volumeMounts:
- mountPath: /mnt/kubeflow-gcfs
name: kubeflow-gcfs
volumes:
- name: kubeflow-gcfs
persistentVolumeClaim:
claimName: kubeflow-gcfs
readOnly: false
```

![8d731664134b224973a790c50a2885d](https://github.com/kenwoodjw/kubeflow-install/assets/39894166/ac55e2ab-5353-4dac-8b2e-79648053ba20)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the PyTorchJob YAML and the official MNIST DDP GPU image named in the issue, then inspect the reported run failure and Kubernetes job or pod status. The issue does not include an error message or logs, so the work is done when the manifest successfully runs the distributed MNIST example.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, pytorch
Domain
infrastructure, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.