common-workflow-language / common-workflow-language/cwltool

`cwltool#MPIRequirement` implementation needed for Singularity

Open
#2,208 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

To use Singularity with MPI we normally use something like `mpirun -np N singularity image.sif args`

Then MPI takes care to communicate with the daemon and ask it to launch N processes using the Singularity container, for the image.sif with the right args, and passing the rank, MPI world size, etc.

That can be in hybrid or host/bind modes. The syntax of both is similar, but in the bind version we need to pass the `--bind` or `--mount` args to add the MPI installation from host inside the container (and have compatible glibc, other libs, maybe some MPI binaries too, etc).

CWLTool fails to run MPI + Singularity in either mode, on my laptop or on HPCs due to the isolation of environment variables and to how it isolates the temporary directory.

Each MPI implementation may use a different set of environment variables. Using `--preserve-entire-environment` doesn't seem to work with containers, and the command line used in Singularity includes `--cleanenv` and `--contain` even when that's used.

The MPI daemon/launcher process will use temporary directories to keep metadata, sockets for IPC, and other files. So when I launch `mpirun` in the host, what I think is happening is `mpirun` creates the set of files/variables needed for the MPI processes, `mpirun -np N singularity` starts separate processes for singularity, but because there's the mount for a different tmpdir, each process launched isn't properly set up for the current group/communicator/etc., and the program crashes.

To confirm that this is happening, I tried this diff against commit

```bash
commit dfc950ca3e4c725fea2807b2f71a4a87c87e3ce4 (HEAD -> main, upstream/main, upstream/HEAD)
Author: Michael R. Crusoe
Date: Tue Feb 24 17:58:52 2026 +0100

ci: pin setuptools/pip version for "no leftovers" test

```

```diff
diff --git a/cwltool/singularity.py b/cwltool/singularity.py
index 41db3b69..2a5a04c8 100644
--- a/cwltool/singularity.py
+++ b/cwltool/singularity.py
@@ -593,9 +593,9 @@ class SingularityCommandLineJob(ContainerCommandLineJob):
"singularity",
"--quiet",
"run" if (is_apptainer_1_1_or_newer() or is_version_3_10_or_newer()) else "exec",
- "--contain",
+ # "--contain",
"--ipc",
- "--cleanenv",
+ # "--cleanenv",
]
if is_apptainer_1_1_or_newer() or is_version_3_10_or_newer():
runtime.append("--no-eval")
@@ -624,9 +624,9 @@ class SingularityCommandLineJob(ContainerCommandLineJob):
writable=True,
)

- self.append_volume(
- runtime, os.path.realpath(self.tmpdir), self.CONTAINER_TMPDIR, writable=True
- )
+ # self.append_volume(
+ # runtime, os.path.realpath(self.tmpdir), self.CONTAINER_TMPDIR, writable=True
+ # )

self.add_volumes(
self.pathmapper,
```

By not isolating the env vars, and by not creating a separate temporary directory, MPI is able to correctly launch the application.

For the env vars, I think we can add something in the docs that tell users to use preserve-entire or to preserve some envs (like everything MPICH_*). But we need to ensure those are passed to the container with `--env` or some other way.

I don't know what to do about the temporary directory. I tried hard-coding in cwltool code to a directory I created `/tmp/mpi`, and launching CWLTool with `TMPDIR=/tmp/mpi cwltool ...`. But that still failed. It might be a bit more difficult to find a fix for this one.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.