microsoft / microsoft/pai

CPU task on GPU cluster

Open
#5,176 1 comment 0 reactions 1 assignee View on GitHub

@yqwang-ms is already working on this.

Since Feb 4, 2021.

Dominant language
JavaScript
Stars
2.7k
Forks
554
Avg merge
20h 42m
Merged PRs (30d)
14

Description

Organization Name:
Advantech

Short summary about the issue/question:
CPU task behavior on GPU cluster

Brief what process you are following:
Did some test on GPU cluster capability

SKU setup in cluster: 1GPU, 3CPU, 90G RAM

  1. Task with 1GPU, 2CPU and 3G RAM: Running
  2. Task with 1CPU and 3G RAM: Running
  3. Task with 2CPU and 4G RAM: Waiting . Log information is listed below:
I1215 02:42:59.219909       6 scheduling_queue.go:908] About to try and schedule pod default/5f7c9b16f10dc398a943e8aea61d534b-jupyter-0
I1215 02:42:59.219969       6 scheduler.go:453] Attempting to schedule pod: default/5f7c9b16f10dc398a943e8aea61d534b-jupyter-0
I1215 02:42:59.272955       6 factory.go:647] Unable to schedule default/5f7c9b16f10dc398a943e8aea61d534b-jupyter-0: no fit: 0/5 nodes are available: 1 Pod is waiting for preemptible or free resource to appear: insufficient capacity in VC devgpu, 1 node(s) didn't match node selector.; waiting
I1215 02:42:59.273003       6 factory.go:742] Updating pod condition for default/5f7c9b16f10dc398a943e8aea61d534b-jupyter-0 to (PodScheduled==False, Reason=Unschedulable)
I1215 02:42:59.275557       6 generic_scheduler.go:1118] Node testcpu0-1111972-iaas is a potential node for preemption.
I1215 02:42:59.275574       6 generic_scheduler.go:1118] Node testgpu0-1111945-iaas is a potential node for preemption.
I1215 02:42:59.275577       6 generic_scheduler.go:1118] Node testgpu1-1112050-iaas is a potential node for preemption.
I1215 02:42:59.275580       6 generic_scheduler.go:1118] Node testcpu1-1111975-iaas is a potential node for preemption.

Why the different behavior for dry-run No. 2 and No.3 ?

How to reproduce it:
Assign task with 0 GPU and multiple CPU to GPU cluster.

OpenPAI Environment:

  • OpenPAI version:
    v1.0.1
  • Cloud provider or hardware configuration:
    OpenStack VM
  • OS (e.g. from /etc/os-release):
    ubuntu 16.04.6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.