aws / aws/sagemaker-python-sdk
PySparkProcessor - Possibility to choose different instance types for the driver node and the worker nodes
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 32
Description
**Describe the feature you'd like**
It would be nice to have the possibility to choose different instance types for the driver node and the worker nodes when using the `PySparkProcessor`.
**How would this feature be used? Please describe.**
```
pyspark_processor = PySparkProcessor(
base_job_name=...,
framework_version=...,
role=...,
driver_instance_type="ml.m5.4xlarge",
worker_instance_type="ml.m5.large",
instance_count=...,
sagemaker_session=pipeline_session,
max_runtime_in_seconds=...,
)
```
**Describe alternatives you've considered**
It is possible to choose a high-memory instance for all instances but it could be unnecessarily costly for the user.
**Additional context**
Some pyspark operations (_e.g_. `.toPandas()`) are memory expensive for the driver node.
Contributor guide
Assessment
This issue has not been assessed yet.