aws / aws/amazon-sagemaker-examples

[Bug Report] SageMaker Pipelines Notebook - XGBoost is first installed with anaconda then upgraded with pip.

Open
#4,779 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/cddb473cc79c2eaae5d7fb79c456280cc5d6471d/%20%20%20ml_ops/sm-pipelines_batch_inference_step_decorator/sm-pipelines_batch_inference_step_decorator.ipynb

**Describe the bug**
Pipeline execution fails at Training Step, due to two different version installations of XGboost. One by Conda from the Sagemaker Distribution Image. The other from Pip in the requirements.txt.

**To reproduce**
Run the notebook on SageMaker Studio. Distribution Image 2.1.0.

**Potential Fix identified**
Works if we run on SageMaker Distribution Image 1.11. However this is not the default anymore, so customer will run into this issue more often.

1. Update XGboost in requirements.txt to 2.1.1.
2. Move `early_stopping_rounds=5` from `xgb.fit()` to `XGBClassifier()`
Like below:
```
xgb = XGBClassifier(n_estimators=num_round, early_stopping_rounds=5, **param)
xgb.fit(
train_df,
y_train,
eval_set=[(validation_df, y_validation)],
)
```
3. Now need to fix Evaluate step. Errors with ModelBuilder.

**Logs**

>
>
> Traceback (most recent call last): File "/opt/conda/lib/python3.11/site-packages/sagemaker/remote_function/invoke_function.py", line 144, in main _execute_remote_function( File "/opt/conda/lib/python3.11/site-packages/sagemaker/remote_function/invoke_function.py", line 119, in _execute_remote_function stored_function.load_and_invoke() File "/opt/conda/lib/python3.11/site-packages/sagemaker/remote_function/core/stored_function.py", line 189, in load_and_invoke result = func(*resolved_args, **resolved_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/ipykernel_1942/1143899264.py", line 26, in train File "/opt/conda/lib/python3.11/site-packages/xgboost/__init__.py", line 7, in from . import collective, dask, rabit File "/opt/conda/lib/python3.11/site-packages/xgboost/collective.py", line 12, in from .core import _LIB, _check_call, c_str, py_str, from_pystr_to_cstr File "/opt/conda/lib/python3.11/site-packages/xgboost/core.py", line 264, in _LIB = _load_lib() ^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/xgboost/core.py", line 258, in _load_lib raise ValueError(msg)
>
>
> ValueError: Mismatched version between the Python package and the native shared object. Python package version: 1.7.1. Shared object version: 2.1.1. Shared object is loaded from: /opt/conda/lib/libxgboost.so.
> Likely cause: * XGBoost is first installed with anaconda then upgraded with pip. To fix it please remove one of the installations.

**After Doing steps 1 and 2 of the fix above, get this error in the Evaluate step.**
> ValueError: Unable to auto detect a DLC for framework xgboost, framework version py311 and python version 2.1.1. Please manually provide image_uri to ModelBuilder()

Note: I am an AWS employee. Please feel free to message internally.

Contributor guide

Open the contributing guide

Research direction

Start with the linked ml_ops/sm-pipelines_batch_inference_step_decorator/sm-pipelines_batch_inference_step_decorator.ipynb and its requirements.txt, then run the notebook on SageMaker Studio with Distribution Image 2.1.0. Check the Training Step after the XGBoost changes and investigate the Evaluate step's ModelBuilder image_uri error. Done means the pipeline completes through evaluation without the reported version mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
anaconda, aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.