Azure / Azure/cyclecloud-slurm

Problem using the default image

Open
#28 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
84
Forks
56
Avg merge
3d 17h
Merged PRs (30d)
1

Description

I'm not sure if I'm doing something wrong, but I'm utterly failing to use the default Slurm image to do some basic MPI.

I've stood up a simple cyclecloud-slurm setup with the default image (Cycle CentOS 7) for all node types.

I can run a simple single node MPI job as long as I use Intel MPI:

srun -n2 mpirun ./hello

That works fine.

But as soon as I try to use more than one node, I get all sorts of infiniband related error messages.

If I try to use a machinetype that supports Infiniband I end up with no functioning MPI at all, as it appears to not get installed.

Try even a single node MPI test with OpenMPI and I get errors possibly related to hostname issues:

```
--------------------------------------------------------------------------
ORTE has lost communication with a remote daemon.

HNP daemon : [[22239,0],0] on node ip-0A000406
Remote daemon: [[22239,0],1] on node hpc-1
```

Looks like OpenMPI is working with the Slurm node name (hpc-1) rather than that actual hostname (ip-0A000406), and then possibly getting upset as a result?

If I've missed something really obvious, please do point it out :)

Thanks,

John

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.