Azure / Azure/cyclecloud-gridengine

SGE configuration resulting in memory blocked jobs in H-series nodes

Offen
#21 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @mvrequa Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
7
Forks
18
Ø Merge
5 T. 21 Std.
Gemergte PRs (30 T.)
2

Beschreibung

**Cyclecloud version: 8.1.0-1275**
**OS Image: HPC:CentOS8.1**
**Problem: Issues with Infiniband connectivity**

I was running a small cluster in cyclecloud 7 and recently I updated to cyclecloud 8, the cluster has several node arrays, some non H-series and other [H-series](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-hpc) for MPI heavy calculations. I noticed that there seems to be a different behaviour in the Infiniband connectivity between 7 and 8. I have tracked part of the problem to the SGE configuration.

The SGE configuration provided in cyclecloud 8.1.0-1275 with the [HPC:CentOS8.1](https://docs.microsoft.com/en-us/azure/cyclecloud/how-to/hb-hc-best-practices?view=cyclecloud-8) has the variable `execd_params` set to `none`. This can result in problems when trying to execute MPI jobs in H-series nodes with Infiniband.
The jobs would fail reporting that the the process cannot allocate the necessary memory, this issue can be [manually fixed](http://arc.liv.ac.uk/pipermail/gridengine-users/2009-January/022752.html) by setting `execd_params H_MEMORYLOCKED=unlimited`.
I have tested this using several MPI heavy programs such as [VASP](https://www.vasp.at/) and [QE ](https://www.quantum-espresso.org/), which were running in non H-series nodes without problem, but when trying to run in H-series would consistently fail. After the change of `execd_params` the programs would run.

However, something I noticed from updating to cyclecloud8 from cyclecloud7 is that running MPI jobs in non H-series jobs using inter node communication seems to no longer be possible.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.