aws / aws/sagemaker-python-sdk

PySparkProcessor - Possibility to choose different instance types for the driver node and the worker nodes

Offen
#3,616 1 Kommentar 0 Reaktionen 1 zugewiesene Person Beansprucht von @nargokul Auf GitHub ansehen
component: processing PySpark
Vorherrschende Sprache
Python
Sterne
2.3k
Forks
1.3k
Ø Merge
1 T. 22 Std.
Gemergte PRs (30 T.)
35

Beschreibung

**Describe the feature you'd like**
It would be nice to have the possibility to choose different instance types for the driver node and the worker nodes when using the `PySparkProcessor`.

**How would this feature be used? Please describe.**
```
pyspark_processor = PySparkProcessor(
base_job_name=...,
framework_version=...,
role=...,
driver_instance_type="ml.m5.4xlarge",
worker_instance_type="ml.m5.large",
instance_count=...,
sagemaker_session=pipeline_session,
max_runtime_in_seconds=...,
)
```

**Describe alternatives you've considered**
It is possible to choose a high-memory instance for all instances but it could be unnecessarily costly for the user.

**Additional context**
Some pyspark operations (_e.g_. `.toPandas()`) are memory expensive for the driver node.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.