aws / aws/amazon-sagemaker-examples
[Bug Report] SageMaker Pipelines Notebook - XGBoost is first installed with anaconda then upgraded with pip.
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/cddb473cc79c2eaae5d7fb79c456280cc5d6471d/%20%20%20ml_ops/sm-pipelines_batch_inference_step_decorator/sm-pipelines_batch_inference_step_decorator.ipynb
**Describe the bug**
Pipeline execution fails at Training Step, due to two different version installations of XGboost. One by Conda from the Sagemaker Distribution Image. The other from Pip in the requirements.txt.
**To reproduce**
Run the notebook on SageMaker Studio. Distribution Image 2.1.0.
**Potential Fix identified**
Works if we run on SageMaker Distribution Image 1.11. However this is not the default anymore, so customer will run into this issue more often.
1. Update XGboost in requirements.txt to 2.1.1.
2. Move `early_stopping_rounds=5` from `xgb.fit()` to `XGBClassifier()`
Like below:
```
xgb = XGBClassifier(n_estimators=num_round, early_stopping_rounds=5, **param)
xgb.fit(
train_df,
y_train,
eval_set=[(validation_df, y_validation)],
)
```
3. Now need to fix Evaluate step. Errors with ModelBuilder.
**Logs**
>
>
> Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/sagemaker/remote_function/invoke_function.py", line 144, in main _execute_remote_function( File "/opt/conda/lib/python3.11/site-packages/sagemaker/remote_function/invoke_function.py", line 119, in _execute_remote_function stored_function.load_and_invoke() File "/opt/conda/lib/python3.11/site-packages/sagemaker/remote_function/core/stored_function.py", line 189, in load_and_invoke result = func(*resolved_args, **resolved_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/ipykernel_1942/1143899264.py", line 26, in train File "/opt/conda/lib/python3.11/site-packages/xgboost/__init__.py", line 7, in from . import collective, dask, rabit File "/opt/conda/lib/python3.11/site-packages/xgboost/collective.py", line 12, in from .core import _LIB, _check_call, c_str, py_str, from_pystr_to_cstr File "/opt/conda/lib/python3.11/site-packages/xgboost/core.py", line 264, in _LIB = _load_lib() ^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/xgboost/core.py", line 258, in _load_lib raise ValueError(msg)
>
>
> ValueError: Mismatched version between the Python package and the native shared object. Python package version: 1.7.1. Shared object version: 2.1.1. Shared object is loaded from: /opt/conda/lib/libxgboost.so.
> Likely cause: * XGBoost is first installed with anaconda then upgraded with pip. To fix it please remove one of the installations.
**After Doing steps 1 and 2 of the fix above, get this error in the Evaluate step.**
> ValueError: Unable to auto detect a DLC for framework xgboost, framework version py311 and python version 2.1.1. Please manually provide image_uri to ModelBuilder()
Note: I am an AWS employee. Please feel free to message internally.
Contributor guide
Research direction
Start with the linked ml_ops/sm-pipelines_batch_inference_step_decorator/sm-pipelines_batch_inference_step_decorator.ipynb and its requirements.txt, then run the notebook on SageMaker Studio with Distribution Image 2.1.0. Check the Training Step after the XGBoost changes and investigate the Evaluate step's ModelBuilder image_uri error. Done means the pipeline completes through evaluation without the reported version mismatch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- anaconda, aws, jupyter-notebook, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100