AllenInstitute / AllenInstitute/bmtk

Problem importing bmtk.analyzer.compartment

Open
#327 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
335
Forks
102
Avg merge
1d 5h
Merged PRs (30d)
6

Description

I have pip installed BMTK version 1.0.8 on our HPC cluster, running on Rocky8 OS and with Intel Icelake CPUs.
When I start an interactive job with 16 tasks, I fail to import the bmtk.analyzer.compartment package:

$ nproc
16
$ module use /apps/leuven/rocky8/icelake/2022b/modules/all
$ module load BMTK/1.0.8-foss-2022b
$ python
Python 3.10.8 (main, Jul 13 2023, 22:10:28) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bmtk
>>> import bmtk.analyzer.compartment
[m28c27n1:3237025] OPAL ERROR: Unreachable in file ext3x_client.c at line 112
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[m28c27n1:3237025] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

I have built BMTK/1.0.8-foss-2022b (and all its dependencies) against OpenMPI/4.1.4-GCC-12.2.0 module. However, this specific OpenMPI module is not built with Slurm support. That's why parallel applications which are launched using srun would spit out the OPAL error message above.

I would like to ask if there exists an environment variable to choose how the tasks would be launched? So that I can choose to use mpirun directly instead of srun.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the import of bmtk.analyzer.compartment in the reported BMTK/1.0.8-foss-2022b environment, then inspect how the package initializes OpenMPI under Slurm. No source file or test is named; done means the package imports successfully when the cluster uses the supported task launcher.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.