CPU task on GPU cluster
Open
@yqwang-ms is already working on this.
Since Feb 4, 2021.
- Dominant language
- JavaScript
- Stars
- 2.7k
- Forks
- 554
- Avg merge
- 20h 42m
- Merged PRs (30d)
- 14
Description
Organization Name:
Advantech
Short summary about the issue/question:
CPU task behavior on GPU cluster
Brief what process you are following:
Did some test on GPU cluster capability
SKU setup in cluster: 1GPU, 3CPU, 90G RAM
- Task with 1GPU, 2CPU and 3G RAM: Running
- Task with 1CPU and 3G RAM: Running
- Task with 2CPU and 4G RAM: Waiting . Log information is listed below:
I1215 02:42:59.219909 6 scheduling_queue.go:908] About to try and schedule pod default/5f7c9b16f10dc398a943e8aea61d534b-jupyter-0
I1215 02:42:59.219969 6 scheduler.go:453] Attempting to schedule pod: default/5f7c9b16f10dc398a943e8aea61d534b-jupyter-0
I1215 02:42:59.272955 6 factory.go:647] Unable to schedule default/5f7c9b16f10dc398a943e8aea61d534b-jupyter-0: no fit: 0/5 nodes are available: 1 Pod is waiting for preemptible or free resource to appear: insufficient capacity in VC devgpu, 1 node(s) didn't match node selector.; waiting
I1215 02:42:59.273003 6 factory.go:742] Updating pod condition for default/5f7c9b16f10dc398a943e8aea61d534b-jupyter-0 to (PodScheduled==False, Reason=Unschedulable)
I1215 02:42:59.275557 6 generic_scheduler.go:1118] Node testcpu0-1111972-iaas is a potential node for preemption.
I1215 02:42:59.275574 6 generic_scheduler.go:1118] Node testgpu0-1111945-iaas is a potential node for preemption.
I1215 02:42:59.275577 6 generic_scheduler.go:1118] Node testgpu1-1112050-iaas is a potential node for preemption.
I1215 02:42:59.275580 6 generic_scheduler.go:1118] Node testcpu1-1111975-iaas is a potential node for preemption.
Why the different behavior for dry-run No. 2 and No.3 ?
How to reproduce it:
Assign task with 0 GPU and multiple CPU to GPU cluster.
OpenPAI Environment:
- OpenPAI version:
v1.0.1 - Cloud provider or hardware configuration:
OpenStack VM - OS (e.g. from /etc/os-release):
ubuntu 16.04.6
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.