futureverse / futureverse/future.batchtools

steps to speed up job submission?

Open
#36 13 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
R
Stars
87
Forks
10
PR merge metrics
No merged PRs in 30d

Description

I'm using `future.batchtools` via `drake`, and just got my first plan running on the cluster. It seems to take about one minute for each job submitted, and since I'm trying to submit several hundred jobs, that's not ideal (although it's not a deal-breaker, because I expect each job to take many hours to finish). I'm not sure what I might be able to change in order to speed this up. I haven't dived into the code, but my idea of what needs to happen to start a worker is:

1. analyse the code to find dependencies
2. submit the job to the scheduler (SLURM in my case)
3. wait for the job to be allocated
4. wait for the worker to start up (and load libraries?)
5. send data to the worker (and libraries?)

Is this basically accurate?

Does the worker load libraries already installed on its node, or are all libraries sent to the worker by the master? If the latter, then reducing library dependencies seems like a potential avenue to try.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.