aws / aws/amazon-sagemaker-examples

[Bug Report] For XGBoost, confusion and TreeDepth rules are not working.

Open
#2,772 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
Add the link to the notebook.

**Describe the bug**
For XGBoost, confusion and TreeDepth rules are not working. We have tried using 1.2-1 and latest image of XGB.

**To reproduce**
Use any of the following -
sagemaker.image_uris.retrieve("xgboost", region, "1.2-1") OR
sagemaker.image_uris.retrieve("xgboost", region, "latest")
and then used these as rules -
rules = [
Rule.sagemaker(
base_config=rule_configs.confusion(),
rule_parameters={
"category_no": "2",
"labels": "labels",
"predictions": "predictions",
"labels_collection": "labels",
"predictions_collection": "predictions",
"min_diag": "0.9",
"max_off_diag": "0.1"
},
collections_to_save=[
CollectionConfig(
name="labels",
parameters={
"save_interval": str(save_interval)
}
),
CollectionConfig(
name="predictions",
parameters={
"include_regex": str(save_interval)
}
)
]
),
Rule.sagemaker(
base_config=rule_configs.tree_depth(),
rule_parameters={
"depth": "4"
},
collections_to_save=[
CollectionConfig(
name="trees",
parameters={
"save_interval": str(save_interval)
}
)
]
)
]

Using an estimator object.

**Logs**
Checking rule output using,

xgboost_algorithm_mode_estimator.latest_training_job.rule_job_summary()

We got the folowing error for both confusion and tree_depth rule,
'RuleEvaluationStatus': 'Error',
'StatusDetails': 'InternalServerError: We encountered an internal error. Please try again.'

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the shown sagemaker.image_uris.retrieve calls, confusion and tree_depth rules, and estimator rule_job_summary(). Check whether the saved collections and rule parameters produce the reported InternalServerError for both rules. Done means identifying the cause and confirming that both rule evaluations complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.