futureverse / futureverse/future

AWS Batch backend

Open
#423 6 comments 4 reactions 0 assignees View on GitHub
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

I propose [AWS Batch](https://aws.amazon.com/batch/) as a new `clustermq` scheduler. Batch has become extremely popular, especially as traditional HPC is waning. I have a strong personal interest in making Batch integrate nicely with R (ref: https://github.com/wlandau/targets/issues/152, https://github.com/wlandau/tarchetypes/issues/8, https://wlandau.github.io/targets-manual/cloud.html).

Batch is super easy to set up through the AWS web console, and I think it would fit nicely into `future`'s ecosystem: maybe with something like `future::plan(future.aws.batch::future_aws_batch, template = "batch.tmpl")`, where `batch.tmpl` contains an [AWS API call](https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html) with the compute environment, job queue, job definition, and key pair. I think we could use [`curl`](https://github.com/jeroen/curl) directly instead of the much larger and rapidly developing [`paws`](https://github.com/paws-r/paws) package. The tricky part is how we retrieve the data back from an AWS Batch job. I'm not sure how to do that yet.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.