[Documentation] Call shell scripts with extra resources inside the submitted Python function
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77
- Forks
- 7
- Avg merge
- 10h 32m
- Merged PRs (30d)
- 12
Description
For SLURM each call of srun creates a new job step, so there is no need to assign any resources to the specific Python function, just assign the resources to the external shell script directly.
For flux there is the option to use nested executors:
https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#nested-executors
Still this option can also be assigned during the submission - just like all the other options:
from executorlib import FluxJobExecutor
def get_available_gpus(lst):
import socket
from tensorflow.python.client import device_lib
local_device_protos = device_lib.list_local_devices()
return [
(x.name, x.physical_device_desc, socket.gethostname())
for x in local_device_protos if x.device_type == "GPU"
] + lst
with FluxJobExecutor() as exe:
fs = []
for i in range(1, 4):
fs = exe.submit(
get_available_gpus,
lst=fs,
resource_dict={"cores": 1, "gpus_per_core": 1, "flux_executor_nesting": True},
)
print(fs.result())
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked nested-executors documentation and the repository’s existing HPC job documentation; no specific file is named in the issue. Document how SLURM shell scripts and Flux nested executors receive extra resources at submission time, using the shown Python example as the reference for what done looks like.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- documentation, hpc
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100