Azure / Azure/cyclecloud-slurm-workspace
Persistence Mode not set for GPU
- Dominant language
- Bicep
- Stars
- 10
- Forks
- 18
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 4
Description
In CCWS version 2025.12.01 which uses CC 8.8.1, a new healthagent is always on and will monitor GPU and IB. For old GPU SKUs the following error message can be raised: `Persistence Mode not set for GPU x`
This is because recent GPU SKUs have this mode enabled and older may not.
# How to Fix
This can be fixed by enabling the persistent mode in the cloud-init of the target partition using the `nvidia-smi` CLI.
```
#!/bin/bash
nvidia-smi -pm 1
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the target partition's cloud-init configuration in the deployment files and trace how GPU initialization is defined. Apply the reported nvidia-smi persistence-mode change there, then verify on an older GPU SKU that the healthagent no longer reports that persistence mode is unset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, shell
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100