futureverse / futureverse/future
Support for auto-scaling workloads via Kubernetes
- Dominant language
- R
- Stars
- 1k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
Hi Henrik, as started on Twitter...
I'd like to use `future` on jobs without a predefined number of cores, on an [autoscaling Kubernetes cluster](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler).
These work by abstracting job loads into pods, which then run abstracted workloads that would be R futures on a cluster wherever there is room for it on the cluster e.g. if a cluster has 3 nodes with 4 CPUs each, then you have 12 CPU cores you can use to plan.
The cool bit is there is an option for autoscaling, so once a node hits CPU or RAM limits, it can launch a new node with the same configuration, which future 'futures' can take advantage of. This would mean a dynamic number of CPUs available, but how could this work with `future`?
This looks close to what this means in a `future` context:
> automatic rescaling of worker set without calling plan(), e.g. each time a future is created/launched? Basically a internal, "continuous" refresh of plan(..., workers = ...)
...so I guess there would need to be some API call that returns the number of CPUs available on the cluster?
I'm looking to do this via [googleKubernetesR](https://github.com/RhysJackson/googleKubernetesR), but I'm sure the same principle would apply to the other cloud providers too that all now support kubernetes.
Contributor guide
Assessment
This issue has not been assessed yet.