Azure / Azure/cyclecloud-scalelib

Incorrect VM count reported for HC/NC machines

Open
#27 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
16
Avg merge
3d 16h
Merged PRs (30d)
1

Description

In CycleCloud 8.5 release, which should ship with updated scalelib, I'm getting incorrect available VM count in `azpbs buckets` output (or `azslurm buckets`).

Seems to not affect F2 series machines, however affects HC44 and NCv4 as can be seen here:
I have 440 HC cores available which correspond to 10 HC44rs VMs. `azpbs buckets` shows there are only two available:
![Screenshot 2024-01-10 132506](https://github.com/Azure/cyclecloud-scalelib/assets/1652482/e9e1badf-ef98-46df-a9bb-84cd9cef40a4)

I have 480 NCv4 cores available which correspond to 5 NC96ads_A100_v4 VMs. `azpbs buckets` shows there is only one available:
![Screenshot 2024-01-10 134119](https://github.com/Azure/cyclecloud-scalelib/assets/1652482/2d8a127e-6d0e-422c-ba1f-2ba3a7fb1780)

Reverting to a template with older scalelib fixes the issue.
This issue is reproducible with both PBS and Slurm, and affects job scheduling as cluster can no longer scale beyond 2 nodes on these VM SKUs.
Is this a known issue?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the count discrepancy with `azpbs buckets` and `azslurm buckets` using the HC44 and NCv4 examples, then compare behavior with the older scalelib template that reportedly works. Done means the available VM counts match the available cores for both SKU families and scheduling can scale beyond the reported node limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.