[Documentation] Worker for executing multiple Python functions
Open
Nobody has claimed this yet.
documentation
- Dominant language
- Python
- Stars
- 77
- Forks
- 7
- Avg merge
- 10h 32m
- Merged PRs (30d)
- 12
Description
There are basically three ways to use a Python Task Managers:
- Submit each Python Function as an individual job to the HPC cluster - supported in executorlib by the
SlurmClusterExecutorand theFluxClusterExecutor. - Submit multiple Python Functions within one HPC job allocation - supported in executorlib by the
SlurmJobExecutorand theFluxJobExecutor. - Submit multiple Python Functions to the same resources on a given HPC cluster - similar to the
block_allocation=Truefor theSingleNodeExecutor, theSlurmJobExecutorand theFluxJobExecutorbut for theSlurmClusterExecutorand theFluxClusterExecutor. This is the standard use case for solutions like https://www.dask.org and https://parsl.readthedocs.io/ so it might be an interesting addition to executorlib.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no file or test. Start by comparing SlurmClusterExecutor and FluxClusterExecutor with SlurmJobExecutor and FluxJobExecutor, including the existing block_allocation behavior described in the issue. Clarify whether the goal is documentation or a new worker capability, then define completion around executing multiple Python functions on shared cluster resources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100