[feature request] maximizeResourceAllocation option
Open
- Dominant language
- Python
- Stars
- 69
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
Spark on EMR has a [`maximizeResourceAllocation`](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-configure.html#emr-spark-maximizeresourceallocation) option which makes it really easy to run a Spark Job with all of the cluster's resources. This uses the approach of creating "fat" workers (less workers with more resources).
An older version of this implementation can be found here: https://github.com/aws-samples/emr-bootstrap-actions/blob/6065841329f81d00e402df909bd69fff9af6e62e/spark/maximize-spark-default-config
It would be great to have a similar option when using Dask on EMR so the user has to do less custom configuration.
Contributor guide
Assessment
This issue has not been assessed yet.