aws / aws/sagemaker-spark

Batch transform job using sagemaker_pyspark

Open
#107 1 comment 2 reactions 0 assignees View on GitHub
feature request
Dominant language
Scala
Stars
301
Forks
129
PR merge metrics
No merged PRs in 30d

Description

Please fill out the form below.

### System Information
- **PySpark**:
- **latest**:
- **2.3.2**:
- **XGBoost**:

### Describe the problem
Are you guys planning on implementing the [Transformer](https://sagemaker.readthedocs.io/en/stable/transformer.html) class in sagemaker_pyspark? I know I am able to use SageMakerModel class to create an endpoint and transform a spark dataframe. It would also be useful to be able to transform a spark dataframe using a batch transform job without me having to worry about deleting the endpoint. Thank you.

### Minimal repo / logs

- **Exact command to reproduce**:

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing SageMakerModel entry point and the linked Transformer documentation, then trace how Spark dataframes are currently sent to endpoints. Define the batch transform job flow and verify that a dataframe can be transformed without requiring an endpoint or manual endpoint deletion.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, scala, spark
Domain
cloud, data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.