Azure / Azure/cyclecloud-gridengine

SGE configuration resulting in memory blocked jobs in H-series nodes

オープン
#21 コメント 0 件 リアクション 0 件 担当者 1 名 @mvrequa が担当を希望しています GitHub で見る
主要言語
Python
スター
7
フォーク
18
平均マージ
5日 21時間
マージ済み PR(30日)
2

説明

**Cyclecloud version: 8.1.0-1275**
**OS Image: HPC:CentOS8.1**
**Problem: Issues with Infiniband connectivity**

I was running a small cluster in cyclecloud 7 and recently I updated to cyclecloud 8, the cluster has several node arrays, some non H-series and other [H-series](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-hpc) for MPI heavy calculations. I noticed that there seems to be a different behaviour in the Infiniband connectivity between 7 and 8. I have tracked part of the problem to the SGE configuration.

The SGE configuration provided in cyclecloud 8.1.0-1275 with the [HPC:CentOS8.1](https://docs.microsoft.com/en-us/azure/cyclecloud/how-to/hb-hc-best-practices?view=cyclecloud-8) has the variable `execd_params` set to `none`. This can result in problems when trying to execute MPI jobs in H-series nodes with Infiniband.
The jobs would fail reporting that the the process cannot allocate the necessary memory, this issue can be [manually fixed](http://arc.liv.ac.uk/pipermail/gridengine-users/2009-January/022752.html) by setting `execd_params H_MEMORYLOCKED=unlimited`.
I have tested this using several MPI heavy programs such as [VASP](https://www.vasp.at/) and [QE ](https://www.quantum-espresso.org/), which were running in non H-series nodes without problem, but when trying to run in H-series would consistently fail. After the change of `execd_params` the programs would run.

However, something I noticed from updating to cyclecloud8 from cyclecloud7 is that running MPI jobs in non H-series jobs using inter node communication seems to no longer be possible.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。