Azure / Azure/cyclecloud-gridengine

SGE configuration resulting in memory blocked jobs in H-series nodes

Open
#21 0 comments 0 reactions 1 assignee Claimed by @mvrequa View on GitHub
Dominant language
Python
Stars
7
Forks
18
Avg merge
5d 21h
Merged PRs (30d)
2

Description

**Cyclecloud version: 8.1.0-1275**
**OS Image: HPC:CentOS8.1**
**Problem: Issues with Infiniband connectivity**

I was running a small cluster in cyclecloud 7 and recently I updated to cyclecloud 8, the cluster has several node arrays, some non H-series and other [H-series](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-hpc) for MPI heavy calculations. I noticed that there seems to be a different behaviour in the Infiniband connectivity between 7 and 8. I have tracked part of the problem to the SGE configuration.

The SGE configuration provided in cyclecloud 8.1.0-1275 with the [HPC:CentOS8.1](https://docs.microsoft.com/en-us/azure/cyclecloud/how-to/hb-hc-best-practices?view=cyclecloud-8) has the variable `execd_params` set to `none`. This can result in problems when trying to execute MPI jobs in H-series nodes with Infiniband.
The jobs would fail reporting that the the process cannot allocate the necessary memory, this issue can be [manually fixed](http://arc.liv.ac.uk/pipermail/gridengine-users/2009-January/022752.html) by setting `execd_params H_MEMORYLOCKED=unlimited`.
I have tested this using several MPI heavy programs such as [VASP](https://www.vasp.at/) and [QE ](https://www.quantum-espresso.org/), which were running in non H-series nodes without problem, but when trying to run in H-series would consistently fail. After the change of `execd_params` the programs would run.

However, something I noticed from updating to cyclecloud8 from cyclecloud7 is that running MPI jobs in non H-series jobs using inter node communication seems to no longer be possible.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.