Azure / Azure/cyclecloud-slurm
Scheduler running out of memory when launching a larger (100+) number of nodes simultaneously
- Dominant language
- Python
- Stars
- 84
- Forks
- 56
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 1
Description
CycleCloud version: 8.6.2-3276
Slurm version: 22.05.11
AFAIK, CycleCloud's prolog script calls `get_acct_info.sh` which calls `azslurm accounting_info` and this happens for each launched node. I'm observing that each launch of `azslurm accounting_info` takes ~150MB of memory, so when launching hundreds of nodes simultaneously, the scheduler can easily get out of memory.
Currently I'm mitigating by commenting out the call to `get_acct_info.sh` in the prolog script.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the prolog script's call to get_acct_info.sh and its azslurm accounting_info invocation, then reproduce launches of 100+ nodes while observing scheduler memory. Done means concurrent launches no longer cause the scheduler to run out of memory, with the accounting behavior addressed or an intentional alternative documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, shell
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100