POSYDON-code / POSYDON-code/POSYDON
[FEATURE]: Add a multiprocessing backend for local parallel population runs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 45
- Forks
- 37
- Avg merge
- 8d 20h
- Merged PRs (30d)
- 5
Description
Is your feature request related to a problem? Please describe.
Currently, running a large POSYDON population efficiently is primarily geared toward HPC systems using SLURM job arrays. This works well on clusters with a scheduler, but is inconvenient for users who have access to a single large shared-memory machine with many CPU cores and sufficient RAM, but no workload manager.
In this situation, running PopulationRunner locally uses only a single process by default, leaving most of the available CPU resources unused.
There is also an existing MPI execution path (use_MPI), but its current maintenance and testing status is unclear. In addition, it requires an MPI installation and launching the calculation through mpiexec, making it more complicated than necessary for a single shared-memory machine.
Describe the solution you would like
Add a scheduler-independent local multiprocessing mode to the population synthesis runner, using Python's multiprocessing infrastructure or concurrent.futures.ProcessPoolExecutor.
For example, the user could specify something equivalent to:
n_processes = 32
or select a local multiprocessing backend, and PopulationRunner would automatically distribute the binaries across the requested number of processes.
Ideally, the implementation should:
- split the binary indices across the worker processes;
- preserve deterministic/reproducible random-number generation;
- allow each worker to write independent temporary/batch files, avoiding concurrent writes to the same HDF5 file;
- automatically combine the worker outputs into the normal POSYDON population file when the run finishes;
- preserve the existing serial behaviour when only one process is requested;
- expose a simple user-facing interface without requiring SLURM, MPI, or another external scheduler.
Much of the existing index-splitting and temporary-file infrastructure used by the SLURM/MPI paths may potentially be reusable for this backend.
An important implementation consideration is memory usage. Each worker may need its own copy of the interpolation/grid objects, so the number of processes should remain explicitly user-configurable rather than automatically using all available cores.
Describe alternatives you have considered
SLURM job arrays already provide an effective solution on managed HPC systems, but they are not available on standalone multi-core machines.
POSYDON also contains an MPI-based local execution path using mpi4py. However, its current testing and maintenance status is unclear, and it requires an MPI runtime and an mpiexec-based workflow.
Users could also manually split a population into multiple independent runs and merge the resulting files afterwards, but this duplicates functionality that POSYDON already implements internally for parallel population runs.
For the common case of a single machine with many CPU cores and substantial RAM, native Python multiprocessing would provide a much simpler solution.
Provide a description of the component of POSYDON involved
This concerns population synthesis, primarily:
PopulationRunnerinposydon.popsyn.synthetic_populationBinaryPopulationinposydon.popsyn.binarypopulation- population-run configuration and output/batch-file handling
The implementation could likely build on the existing mechanisms that divide population indices among ranks/jobs and produce separate temporary output files.
Additional context
This request arose from a user installing POSYDON on their group's large multi-core machine and attempting to run a large population. The machine had enough CPU cores and memory to parallelize the calculation effectively, but no SLURM or other scheduler.
Supporting this type of shared-memory workstation/server directly would make large POSYDON population runs substantially easier outside traditional HPC environments.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with PopulationRunner in posydon.popsyn.synthetic_population and review BinaryPopulation in posydon.popsyn.binarypopulation, focusing on existing index splitting and temporary output handling for SLURM/MPI runs. Define how a configurable local process count preserves serial behavior, reproducible random numbers, and independent batch files. Done means worker outputs are combined into the normal POSYDON population file without requiring SLURM or MPI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100