futureverse / futureverse/future

Support for auto-scaling workloads via Kubernetes

Open
#189 5 comments 3 reactions 0 assignees View on GitHub
feature request
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Hi Henrik, as started on Twitter...

I'd like to use `future` on jobs without a predefined number of cores, on an [autoscaling Kubernetes cluster](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler).

These work by abstracting job loads into pods, which then run abstracted workloads that would be R futures on a cluster wherever there is room for it on the cluster e.g. if a cluster has 3 nodes with 4 CPUs each, then you have 12 CPU cores you can use to plan.

The cool bit is there is an option for autoscaling, so once a node hits CPU or RAM limits, it can launch a new node with the same configuration, which future 'futures' can take advantage of. This would mean a dynamic number of CPUs available, but how could this work with `future`?

This looks close to what this means in a `future` context:

> automatic rescaling of worker set without calling plan(), e.g. each time a future is created/launched? Basically a internal, "continuous" refresh of plan(..., workers = ...)

...so I guess there would need to be some API call that returns the number of CPUs available on the cluster?

I'm looking to do this via [googleKubernetesR](https://github.com/RhysJackson/googleKubernetesR), but I'm sure the same principle would apply to the other cloud providers too that all now support kubernetes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.