[Question] Dynamically change number of concurrent jobs
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 184
- Forks
- 53
- Avg merge
- 7d 2h
- Merged PRs (30d)
- 1
Description
Hi,
Thanks for making this great package. I'm writing code to run multiple registries at the same time. Some tasks (collections of jobs) finish prior to the others. I was wondering if is there any way to release/balance/cancel resources dynamically
For example
- Task 1: job 1,2,3, with concurrent jobs = 2
- Task 2: job 1,2,3,4,5,...,100 with concurrent jobs = 10
- Task 3: job 1-1000
If Task 1 finish earlier than the Task 2, is there a way I can increase the max concurrent jobs to 20 for task 2 and cancel task 3 for those which is still submitting?
One way I come up with is we could monitor certain files within the registry folder, if those files exist, then dynamically change the registry settings for jobs that haven't been submitted yet and then remove those files
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue mentions no files, tests, or entry points. Start by identifying how batchtools represents registries, concurrent jobs, cancellation, and resource limits, then determine the intended behavior for increasing concurrency and stopping pending submissions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100