Azure / Azure/cyclecloud-gridengine
SGE configuration resulting in memory blocked jobs in H-series nodes
- Lingua principale
- Python
- Stelle
- 7
- Fork
- 18
- Merge medio
- 5g 21h
- PR unite (30g)
- 2
Descrizione
**Cyclecloud version: 8.1.0-1275**
**OS Image: HPC:CentOS8.1**
**Problem: Issues with Infiniband connectivity**
I was running a small cluster in cyclecloud 7 and recently I updated to cyclecloud 8, the cluster has several node arrays, some non H-series and other [H-series](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-hpc) for MPI heavy calculations. I noticed that there seems to be a different behaviour in the Infiniband connectivity between 7 and 8. I have tracked part of the problem to the SGE configuration.
The SGE configuration provided in cyclecloud 8.1.0-1275 with the [HPC:CentOS8.1](https://docs.microsoft.com/en-us/azure/cyclecloud/how-to/hb-hc-best-practices?view=cyclecloud-8) has the variable `execd_params` set to `none`. This can result in problems when trying to execute MPI jobs in H-series nodes with Infiniband.
The jobs would fail reporting that the the process cannot allocate the necessary memory, this issue can be [manually fixed](http://arc.liv.ac.uk/pipermail/gridengine-users/2009-January/022752.html) by setting `execd_params H_MEMORYLOCKED=unlimited`.
I have tested this using several MPI heavy programs such as [VASP](https://www.vasp.at/) and [QE ](https://www.quantum-espresso.org/), which were running in non H-series nodes without problem, but when trying to run in H-series would consistently fail. After the change of `execd_params` the programs would run.
However, something I noticed from updating to cyclecloud8 from cyclecloud7 is that running MPI jobs in non H-series jobs using inter node communication seems to no longer be possible.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.