microsoft / microsoft/pai

[pai sku conversion] sku conversion not work on multiple taskrole config without sku config

Open
#5,237 0 comments 0 reactions 1 assignee View on GitHub

@abuccts is already working on this.

Since Jan 11, 2021.

bug
Dominant language
JavaScript
Stars
2.7k
Forks
554
Avg merge
20h 42m
Merged PRs (30d)
14

Description

What would you like to be added:

At multiple taskrole scenery, it will be better to convert each taskrole resource to sku.

Why is this needed:

Without conversion to second taskrole sku, webportal will display strange warning info as following image:
image
And the yaml state is:

protocolVersion: 2
name: horovod_pytorch_78a47051
description: >
  # Horovod PyTorch Job Template

  This is a distributed synthetic benchmark for Horovod with PyTorch backend
  running on OpenPAI.

  It runs [Horovod with Open
  MPI](https://github.com/horovod/horovod/blob/master/docs/mpirun.rst).
contributor: OpenPAI
type: job
jobRetryCount: 0
prerequisites:
  - protocolVersion: 2
    name: horovod_official
    type: dockerimage
    contributor: Horovod
    uri: 'horovod/horovod:0.16.4-tf1.12.0-torch1.1.0-mxnet1.4.1-py3.5'
parameters:
  model: resnet50
  batchsize: 64
  nccl: '-x NCCL_DEBUG=INFO -x NCCL_SOCKET_IFNAME=eth0'
taskRoles:
  master:
    instances: 1
    completion:
      minFailedInstances: 1
      minSucceededInstances: 1
    taskRetryCount: 0
    dockerImage: horovod_official
    resourcePerInstance:
      cpu: 20
      memoryMB: 200000
      gpu: 4
    commands:
      - sleep 10
      - >-
        mpirun --allow-run-as-root -np 8 -H master-0:4,worker-0:4 -bind-to none
        -map-by slot -mca pml ob1 -mca btl ^openib -mca btl_tcp_if_exclude
        lo,docker0 <% $parameters.nccl %> -x PATH -x LD_LIBRARY_PATH python
        pytorch_synthetic_benchmark.py --model <% $parameters.model %>
        --batch-size <% $parameters.batchsize %>
  worker:
    instances: 1
    completion:
      minFailedInstances: 1
    taskRetryCount: 0
    dockerImage: horovod_official
    resourcePerInstance:
      cpu: 16
      memoryMB: 16384
      gpu: 4
    commands:
      - sleep infinity
defaults:
  virtualCluster: default
extras:
  com.microsoft.pai.runtimeplugin:
    - plugin: ssh
      parameters:
        jobssh: true
        sshbarrier: true
  hivedScheduler:
    taskRoles:
      master:
        skuNum: 4
        skuType: K80
      worker:
        skuNum: 4
        skuType: null

This warning can work around by click other taskroles in webportal or fill out sku config for each task role.

Without this feature, how does the current module work:
It shows warning as above.

Components that may involve changes:
I think this is part of webportal function.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.