aws / aws/sagemaker-spark

Example code to use sagemaker-spark for our own model

Open
#137 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
301
Forks
129
PR merge metrics
No merged PRs in 30d

Description

Please fill out the form below.

### System Information
- **Spark or PySpark**: PySpark
- **SDK Version**: Any or most recent
- **Spark Version**: Any or most recent
- **Algorithm (e.g. KMeans)**: Any - in our case, PyTorch models

### Describe the problem
Is there example code of sagemaker-spark to perform inference on a spark dataframe using our own ML model using a GPU inference endpoint spun up by sagemaker? In our case we are bringing over our own PyTorch model.

### Minimal repo / logs
Please provide any logs and a bare minimum reproducible test case, as this will be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
N/A
- **Exact command to reproduce**:
N/A

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing the repository's existing SageMaker Spark examples and inference entry points, then determine how a PySpark DataFrame can use a custom PyTorch model through a SageMaker GPU endpoint; done means a working example covering that flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
data-engineering, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.