aws / aws/sagemaker-python-sdk

Enable passing column type to SHAPConfig in combination with ClarifyCheckStep

未关闭
#5,131 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
component: pipelines type: feature request
主要语言
Python
星标
2.3k
派生
1.3k
平均合并
1 天 22 小时
30 天内合并 PR
35

描述

**Describe the feature you'd like**
Add a parameter to SHAPConfig from sagemaker.workflow.clarify_checkstep which lets the user specify the types of the dataset used to create a baseline for the SHAP analysis (e.g. float, int, category, etc..).
Alternatively, make it possible to run ClarifyCheckStep when an S3 URI has been passed as baseline to SHAPConfig.

**How would this feature be used? Please describe.**
When using the ClarifyCheckStep and SHAPConfig from sagemaker.workflow.clarify_checkstep, I am currently unable to specify my dataset's column types (e.g. some columns should be numerical while others should be categorical).

When running the ClarifyCheckStep as part of a SageMaker pipeline, Clarify calculates a baseline which is erroneous due to not having taken the column types into account, so e.g. some columns that should be categorical gets the mean of the column as baseline, where preferrably they should get the mode of the column or something else more appropriate.

I know that I can pass my own baseline to SHAPConfig, but I don't want this hard coded in my SageMaker pipeline definition - I want it to be computed at runtime, based on previous steps in my SageMaker pipeline.
An alternative solution would be to pass to SHAPConfig the S3 URI to a baseline dataset I create in a previous step, however this doesn't seem to work with how ClarifyCheckStep is currently implemented.

**Describe alternatives you've considered**
Make it possible to run ClarifyCheckStep when an S3 URI has been passed as baseline to SHAPConfig.

**Additional context**
```
from sagemaker.workflow.clarify_check_step import ClarifyCheckStep, ModelExplainabilityCheckConfig, SHAPConfig

shap_config = SHAPConfig(seed=123, num_samples=100, num_clusters=5)

model_explainability_check_config = ModelExplainabilityCheckConfig(
data_config=model_explainability_data_config,
model_config=model_config,
explainability_config=shap_config,
)

step_model_explainability_check = ClarifyCheckStep(
name="ModelExplainabilityCheckStep",
display_name="Model Explainability Check",
clarify_check_config=model_explainability_check_config,
check_job_config=check_job_config_clarify,
skip_check=skipCheckModelExplainabilityParam,
register_new_baseline=registerNewBaselineModelExplainabilityParam,
supplied_baseline_constraints=suppliedBaselineConstraintsModelExplainabilityParam,
model_package_group_name=model_package_group_name,
)
```

贡献指南

打开贡献指南

调研方向

从 sagemaker.workflow.clarify_check_step.py 开始,阅读 SHAPConfig 和 ClarifyCheckStep,重点关注 baseline 值和 S3 URI 的处理方式。比较两种提议的方法——列类型参数或运行时 S3 baseline 支持——并将完成定义为允许 ClarifyCheckStep 在管道中计算或使用类型感知的 SHAP baseline。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
backend-api-design, machine-learning
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。