Azure / Azure/cyclecloud-slurm
Unable to use a custom image as HPC OS
- Dominant language
- Python
- Stars
- 84
- Forks
- 56
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 1
Description
Hi,
I'm trying to use a custom image as the OS for my compute nodes because this allows me to pre-configure software requirements for my compute workflow. I have tried numerous approaches to achieve this, however all have failed.
1. I created a custom vm image based off the Ubuntu-HPC 20.04 - x64 Gen2 available on marketplace. Using this as the OS for compute nodes (HPC partition) failed during boot of the nodes, giving error 'Failed to install jetpack'.
3. I created a new VM based off 1), but this time ensured that Java 8 was set as default (previously it was set to 11). This failed for the same reason
4. I created a new VM based off 2), but this time did a [manual installation of jetpack 8.7.](https://learn.microsoft.com/en-us/azure/cyclecloud/how-to/install-jetpack?view=cyclecloud-8) I ensured jetpack was available to all users by modifying `sudo nano /etc/environment` and appending the following to the PATH `:/opt/cycle/jetpack/bin`. I then re-configured the slurm cluster to ensure that DownloadJetpack = false. This still fails and suggests that Jetpack needs to be installed for this flag to work.
I then added InstallJetpack = false to the configuration and tried again, however this time the compute node gets stuck permanently in 'Awaiting system boot up'.
I can SSH into the compute instance that's part of the HPC scaleset, and when I run jetpack -v from the cyclecloud user I see jetpack is installed with version 8.7.
I'm a bit lost as to what to try next to try and get this to work. Any advice would be appreciated.
Is there anyway to access more detailed logs about the errors in node creation?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.