aws / aws/sagemaker-spark

SagemakerEstimator in Spark ML Pipeline issue

Open
#98 1 comment 1 reaction 0 assignees View on GitHub
bug feature request question
Dominant language
Scala
Stars
301
Forks
129
PR merge metrics
No merged PRs in 30d

Description

Reference: MLFW-2726

System Information
- **Spark**
- **SDK Version**:spark_2.2.0-1.2.5
- **Spark Version**:2.4.3
- **Algorithm**: XGBoost

### Describe the problem
Hi,
I tried to add a sagemakerEstimator within a **Spark ML Pipeline** and fit the training dataset on the pipeline which worked without any issues. When I tried to save the pipeline itself, it threw an exception stating the pipeline contains a stage that is not **writable**.
Is it intended to be that way since when **fit** runs on the sagemakerEstimator, it automatically persists the model to trainingOutputS3DataPath ?
If I wish to have a pipeline persisted which contains other transformer stages along with the sagemakerEstimator instance how would I do it?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Spark ML Pipeline save failure with SagemakerEstimator using Spark 2.4.3 and SDK version spark_2.2.0-1.2.5. Read the pipeline persistence and estimator implementation to determine whether the stage should be writable and define the supported way to persist a pipeline containing it.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, scala, spark
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.