Azure / Azure/cyclecloud-slurm

Scheduler running out of memory when launching a larger (100+) number of nodes simultaneously

Open
#260 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
84
Forks
56
Avg merge
3d 17h
Merged PRs (30d)
1

Description

CycleCloud version: 8.6.2-3276
Slurm version: 22.05.11

AFAIK, CycleCloud's prolog script calls `get_acct_info.sh` which calls `azslurm accounting_info` and this happens for each launched node. I'm observing that each launch of `azslurm accounting_info` takes ~150MB of memory, so when launching hundreds of nodes simultaneously, the scheduler can easily get out of memory.

Currently I'm mitigating by commenting out the call to `get_acct_info.sh` in the prolog script.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the prolog script's call to get_acct_info.sh and its azslurm accounting_info invocation, then reproduce launches of 100+ nodes while observing scheduler memory. Done means concurrent launches no longer cause the scheduler to run out of memory, with the accounting behavior addressed or an intentional alternative documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, shell
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.