futureverse / futureverse/future
AWS Batch backend
- Dominant language
- R
- Stars
- 1k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
I propose [AWS Batch](https://aws.amazon.com/batch/) as a new `clustermq` scheduler. Batch has become extremely popular, especially as traditional HPC is waning. I have a strong personal interest in making Batch integrate nicely with R (ref: https://github.com/wlandau/targets/issues/152, https://github.com/wlandau/tarchetypes/issues/8, https://wlandau.github.io/targets-manual/cloud.html).
Batch is super easy to set up through the AWS web console, and I think it would fit nicely into `future`'s ecosystem: maybe with something like `future::plan(future.aws.batch::future_aws_batch, template = "batch.tmpl")`, where `batch.tmpl` contains an [AWS API call](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html) with the compute environment, job queue, job definition, and key pair. I think we could use [`curl`](https://github.com/jeroen/curl) directly instead of the much larger and rapidly developing [`paws`](https://github.com/paws-r/paws) package. The tricky part is how we retrieve the data back from an AWS Batch job. I'm not sure how to do that yet.
Contributor guide
Assessment
This issue has not been assessed yet.