SagemakerEstimator in Spark ML Pipeline issue
- Dominant language
- Scala
- Stars
- 301
- Forks
- 129
- PR merge metrics
- No merged PRs in 30d
Description
Reference: MLFW-2726
System Information
- **Spark**
- **SDK Version**:spark_2.2.0-1.2.5
- **Spark Version**:2.4.3
- **Algorithm**: XGBoost
### Describe the problem
Hi,
I tried to add a sagemakerEstimator within a **Spark ML Pipeline** and fit the training dataset on the pipeline which worked without any issues. When I tried to save the pipeline itself, it threw an exception stating the pipeline contains a stage that is not **writable**.
Is it intended to be that way since when **fit** runs on the sagemakerEstimator, it automatically persists the model to trainingOutputS3DataPath ?
If I wish to have a pipeline persisted which contains other transformer stages along with the sagemakerEstimator instance how would I do it?
Contributor guide
Research direction
Start by reproducing the Spark ML Pipeline save failure with SagemakerEstimator using Spark 2.4.3 and SDK version spark_2.2.0-1.2.5. Read the pipeline persistence and estimator implementation to determine whether the stage should be writable and define the supported way to persist a pipeline containing it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, scala, spark
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100