Pegasus workflow management for extremely large jobs (near or above 6 digit task count)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 24
- Forks
- 15
- Avg merge
- 10d 3h
- Merged PRs (30d)
- 1
Description
What would you like to see added?
Also "genetic simulations at scale". More generally, how to scale up arbitrary workflows to many, many CPU hours or tasks.
Examples: https://pegasus.isi.edu/application-showcase/
Blake specific example: https://pegasus.isi.edu/application-showcase/inference-of-human-demographic-history-with-approximate-bayesian-computation/ and https://pegasus.isi.edu/2018/12/10/pegasus-helped-ecology-and-evolutionary-biology-graduate-student-at-the-university-of-arizona-shed-light-on-human-population-history/
Paper: https://www.softxjournal.com/article/S2352-7110(18)30121-3/fulltext
Supplementary data showing when/why this is needed: https://www.softxjournal.com/article/S2352-7110(18)30121-3/fulltext#supplementaryMaterial
Bottom line: Executed 5.1 million CPU hours in 5 months for a single researcher.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository file, test, or entry point is named. Start by reviewing the linked Pegasus application examples, paper, and supplementary data to define the documentation scope; done should explain how this site will document scaling arbitrary workflows to very large task counts and CPU-hour workloads.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation, hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100