aws / aws/sagemaker-python-sdk

PySparkProcessor - Possibility to choose different instance types for the driver node and the worker nodes

Open
#3,616 1 comment 0 reactions 1 assignee Claimed by @nargokul View on GitHub
component: processing PySpark
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
2d 2h
Merged PRs (30d)
32

Description

**Describe the feature you'd like**
It would be nice to have the possibility to choose different instance types for the driver node and the worker nodes when using the `PySparkProcessor`.

**How would this feature be used? Please describe.**
```
pyspark_processor = PySparkProcessor(
base_job_name=...,
framework_version=...,
role=...,
driver_instance_type="ml.m5.4xlarge",
worker_instance_type="ml.m5.large",
instance_count=...,
sagemaker_session=pipeline_session,
max_runtime_in_seconds=...,
)
```

**Describe alternatives you've considered**
It is possible to choose a high-memory instance for all instances but it could be unnecessarily costly for the user.

**Additional context**
Some pyspark operations (_e.g_. `.toPandas()`) are memory expensive for the driver node.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.