pyspark XGBoostSageMakerEstimator fails on .fit()
- Dominant language
- Scala
- Stars
- 301
- Forks
- 129
- PR merge metrics
- No merged PRs in 30d
Description
Please fill out the form below.
### System Information
- **Spark or PySpark**: pyspark
- **SDK Version**: latest (pip version 1.4.2), EMR 5.23.0
- **Spark Version**: 2.4.0
- **Algorithm (e.g. KMeans)**: XGBoost
### Describe the problem
Since version 1.4.2 the pyspark XGBoostSageMakerEstimator wrapper class no longer match the corresponding scala class, producing an error in the pyspark JVM communication (during serialization of the python class) when calling pyspark fit function. Specifically, it looks like the property `lamba` was changed to `lambda_weights` without a corresponding change in the scala class.
https://github.com/aws/sagemaker-spark/pull/135/files#diff-ac899a7e58823fff725d351c8459435bb2f09a9687097cd47d3ec34741eb4156R179
It looks like the 1.4.2 release change also bumps the spark version from 2.2.0 to 2.4.0
I can see a couple of workarounds, downgrading EMR to 5.10.1 which is the latest version that has Spark 2.2.0, but I do not want to do this because EMR 5.10.1 does not have support for Jupyter notebooks (only EMR 5.18.0 has support for Jupyter), and I don't want to run Zeppelin notebooks. Another workaround is to sidestep pyspark completely and just use the scala spark sagemaker integration instead of the pyspark variant.
### Minimal repo / logs
This fails with error
```
Param Param(parent='Identifiable_66065fac1a12', name='lambda', doc='L2 regularization term on weights, increase this value will make model more conservative.') does not belong to Identifiable_66065fac1a12.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sagemaker_pyspark/SageMakerEstimator.py", line 256, in fit
self._transfer_params_to_java()
```
Probably, the pyspark communication with Java fails because the pyspark XGBoostSageMakerEstimator class has changed a property previously named `lamba` to `lambda_weights` in a recent change, but the scala class was not changed accordingly.
- **Exact command to reproduce**:
Start an EMR 5.23.0 cluster with a cluster bootstrap action to `pip install sagemaker_pyspark`. Attach an EMR Notebook (JupyterLabs pyspark kernel) and execute the following notebook code
```
from sagemaker_pyspark import IAMRole
from sagemaker_pyspark.algorithms import XGBoostSageMakerEstimator
region = "eu-west-1"
training_data = (spark.read.format("libsvm").option("numFeatures", "784").load("s3a://sagemaker-sample-data-{}/spark/mnist/train/".format(region)))
model_role_arn = "SOME_ROLE_ARN"
xgboost_estimator = XGBoostSageMakerEstimator(
trainingInstanceType="ml.m4.xlarge",
trainingInstanceCount=1,
endpointInstanceType="ml.m4.xlarge",
endpointInitialInstanceCount=1,
sagemakerRole=IAMRole(model_role_arn))
xgboost_estimator.setObjective('multi:softmax')
xgboost_estimator.setNumRound(25)
xgboost_estimator.setNumClasses(10)
xgboost_model = xgboost_estimator.fit(training_data)
```
Contributor guide
Research direction
Start with the Python XGBoostSageMakerEstimator and its corresponding Scala class, then inspect the parameter change linked from pull request #135. Reproduce the failure with the EMR 5.23.0 PySpark notebook command and trace _transfer_params_to_java() during fit(). Done means the Python and Scala parameter definitions match and the provided estimator.fit(training_data) example completes without the reported Param error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scala
- Domain
- backend-api-design, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100