Azure / Azure/cyclecloud-gridengine
SGE configuration resulting in memory blocked jobs in H-series nodes
- Dominant language
- Python
- Stars
- 7
- Forks
- 18
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 2
Description
**Cyclecloud version: 8.1.0-1275**
**OS Image: HPC:CentOS8.1**
**Problem: Issues with Infiniband connectivity**
I was running a small cluster in cyclecloud 7 and recently I updated to cyclecloud 8, the cluster has several node arrays, some non H-series and other [H-series](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-hpc) for MPI heavy calculations. I noticed that there seems to be a different behaviour in the Infiniband connectivity between 7 and 8. I have tracked part of the problem to the SGE configuration.
The SGE configuration provided in cyclecloud 8.1.0-1275 with the [HPC:CentOS8.1](https://docs.microsoft.com/en-us/azure/cyclecloud/how-to/hb-hc-best-practices?view=cyclecloud-8) has the variable `execd_params` set to `none`. This can result in problems when trying to execute MPI jobs in H-series nodes with Infiniband.
The jobs would fail reporting that the the process cannot allocate the necessary memory, this issue can be [manually fixed](http://arc.liv.ac.uk/pipermail/gridengine-users/2009-January/022752.html) by setting `execd_params H_MEMORYLOCKED=unlimited`.
I have tested this using several MPI heavy programs such as [VASP](https://www.vasp.at/) and [QE ](https://www.quantum-espresso.org/), which were running in non H-series nodes without problem, but when trying to run in H-series would consistently fail. After the change of `execd_params` the programs would run.
However, something I noticed from updating to cyclecloud8 from cyclecloud7 is that running MPI jobs in non H-series jobs using inter node communication seems to no longer be possible.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.