Azure / Azure/cyclecloud-slurm-workspace

Persistence Mode not set for GPU

Open Beginner friendly
#349 4 comments 0 reactions 0 assignees View on GitHub
kind/bug
Dominant language
Bicep
Stars
10
Forks
18
Avg merge
2d 9h
Merged PRs (30d)
4

Description

In CCWS version 2025.12.01 which uses CC 8.8.1, a new healthagent is always on and will monitor GPU and IB. For old GPU SKUs the following error message can be raised: `Persistence Mode not set for GPU x`
This is because recent GPU SKUs have this mode enabled and older may not.

Image

# How to Fix
This can be fixed by enabling the persistent mode in the cloud-init of the target partition using the `nvidia-smi` CLI.

```
#!/bin/bash
nvidia-smi -pm 1
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the target partition's cloud-init configuration in the deployment files and trace how GPU initialization is defined. Apply the reported nvidia-smi persistence-mode change there, then verify on an older GPU SKU that the healthagent no longer reports that persistence mode is unset.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, shell
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.