Unable to run CPU task even if resource is available

Organization Name:
Advantech

Short summary about the issue/question:
Unable to run CPU task even if resource is available

Brief what process you are following:

1. Have the following hivescheduler configuration

hivedscheduler:
  config: |
    physicalCluster:
      skuTypes:
        DT:
          gpu: 1
          cpu: 3
          memory: 90613Mi
        CPU:
          cpu: 1
          memory: 8000Mi
      cellTypes:
        DT-NODE:
          childCellType: DT
          childCellNumber: 4
          isNodeLevel: true
        DT-NODE-POOL:
          childCellType: DT-NODE
          childCellNumber: 2
        CPU-NODE:
          childCellType: CPU
          childCellNumber: 15
          isNodeLevel: true
        CPU-NODE-POOL:
          childCellType: CPU-NODE
          childCellNumber: 2
      physicalCells:
      - cellType: DT-NODE-POOL
        cellChildren:
        - cellAddress: testgpu0-1111945-iaas
        - cellAddress: testgpu1-1112050-iaas
      - cellType: CPU-NODE-POOL
        cellChildren:
        - cellAddress: testcpu0-1111972-iaas
        - cellAddress: testcpu1-1111975-iaas
    virtualClusters:
      devgpu:
        virtualCells:
        - cellType: DT-NODE-POOL.DT-NODE
          cellNumber: 2
      devcpu:
        virtualCells:
        - cellType: CPU-NODE-POOL.CPU-NODE
          cellNumber: 2

2. Submit a job with 2 CPU requirement to CPU cluster

    resourcePerInstance:
      cpu: 2
      memoryMB: 3072
      gpu: 0
      ports:
        jupyter: 1
        tensorboard: 1
        grpc: 1

3. The submitted job keep in waiting status even if the resource is enough. The cpu cluster scheduler log indicates insufficient cpu event
   
   hivedscheduler-ds-devcpu-0 log

I1223 11:04:56.563163       6 scheduling_queue.go:908] About to try and schedule pod default/4e326c15a0d1f4a4b9e1ad9ce1ce1a8c-jupyter-0
I1223 11:04:56.563194       6 scheduler.go:453] Attempting to schedule pod: default/4e326c15a0d1f4a4b9e1ad9ce1ce1a8c-jupyter-0
I1223 11:04:56.622323       6 factory.go:647] Unable to schedule default/4e326c15a0d1f4a4b9e1ad9ce1ce1a8c-jupyter-0: no fit: 0/5 nodes are available: 1 Insufficient cpu, 1 Pod is waiting for preemptible or free resource to appear: affinity group has to be scheduled to some nodes not within K8s suggested nodes: ["testcpu1-1111975-iaas"], 1 node(s) didn't match node selector.; waiting
I1223 11:04:56.622444       6 factory.go:742] Updating pod condition for default/4e326c15a0d1f4a4b9e1ad9ce1ce1a8c-jupyter-0 to (PodScheduled==False, Reason=Unschedulable)
I1223 11:04:56.630586       6 generic_scheduler.go:1118] Node testgpu0-1111945-iaas is a potential node for preemption.
I1223 11:04:56.630600       6 generic_scheduler.go:1118] Node testgpu1-1112050-iaas is a potential node for preemption.
I1223 11:04:56.630604       6 generic_scheduler.go:1118] Node testcpu1-1111975-iaas is a potential node for preemption.
I1223 11:04:56.630607       6 generic_scheduler.go:1118] Node testcpu0-1111972-iaas is a potential node for preemption.

How to reproduce it:
Repeat Step 1 ~ 3

OpenPAI Environment:

• OpenPAI version:
  v1.0.1
• Cloud provider or hardware configuration:
  OpenStack cloud
• OS (e.g. from /etc/os-release):
  ubuntu 16.04.6 LTS