support mechanism to allow magpie jobs & classic MPI jobs to run at the same time
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 198
- Forks
- 51
- PR merge metrics
- No merged PRs in 30d
Description
currently job scripts won't work with this because of configurations like
```
#SBATCH --ntasks-per-node=1
```
In addition, options like
```
#SBATCH --no-kill
```
Present problems, as it makes sense for Hadoop/Spark but may not for MPI jobs (unless they can handle failure with SCR and similar libraries).
Need to determine mechanism to allow variable configuration of CPUs by user (X for Big Data, Y for MPI), then subsequent configuration for CPUs in Hadoop, Spark, etc.
Short term, ignore performance of MPI, but longer term binding to specific CPUs would be important for performance (i.e. MPI binds to CPUs A-B, Big Data gets CPUs C-D), unclear how to do this on Magpie side and Big Data side for the moment.
In addition, must determine mechanism by which user's can submit MPI job. Is launching a .sh script sufficient?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the job scripts and their Slurm configuration, then document how MPI, Hadoop, and Spark jobs are submitted and assigned CPUs. Done means a decided mechanism supports variable CPU configurations and MPI submission, with CPU binding identified as a longer-term requirement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, shell, spark
- Domain
- data-engineering, distributed-systems, hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100