Batch transform job using sagemaker_pyspark
- Dominant language
- Scala
- Stars
- 301
- Forks
- 129
- PR merge metrics
- No merged PRs in 30d
Description
Please fill out the form below.
### System Information
- **PySpark**:
- **latest**:
- **2.3.2**:
- **XGBoost**:
### Describe the problem
Are you guys planning on implementing the [Transformer](https://sagemaker.readthedocs.io/en/stable/transformer.html) class in sagemaker_pyspark? I know I am able to use SageMakerModel class to create an endpoint and transform a spark dataframe. It would also be useful to be able to transform a spark dataframe using a batch transform job without me having to worry about deleting the endpoint. Thank you.
### Minimal repo / logs
- **Exact command to reproduce**:
Contributor guide
Research direction
Start by reviewing the existing SageMakerModel entry point and the linked Transformer documentation, then trace how Spark dataframes are currently sent to endpoints. Define the batch transform job flow and verify that a dataframe can be transformed without requiring an endpoint or manual endpoint deletion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, scala, spark
- Domain
- cloud, data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100