aws / aws/amazon-sagemaker-examples

[Example Request] an example notebook to schedule the quality of predictions of multi-class classification models

Open
#3,199 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Describe the use case example you want to see**
an example notebook to monitor on schedule(e.g., hourly, daily) a sagemaker endpoint that serves a multi-class classification model.

**How would this example be used? Please describe.**
This example would be used to learn the following:
1. different models output probabilities or labels the different way. for an example model that classifies between 5 classes, sklearn-model would output 0 or 1 or 2 or 3 or 4 or 5; while a tensorflow model would output the same thing one-hot-encoded, like [1,0,0,0,0] for 0th class and so on. so, this example code would teach an user how to return the probabilities and predicted labels that way sagemaker expects them.
2. when an endpoint classifies between, for example, 300 classes, it is highly likely that the developer is using custom inference code to also return the predicted class name among those 300 and such type of output would also very likely be json, something like, `{'prediction': 23, 'label': 'xxxxx', 'probabilities': [0.0034, 0.104, ....... 0.501, ..... 0.0009]}`. so, readers would know how to mention the json-paths for predictions and label properly in [EndpointInput](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_monitor.model_monitoring.EndpointInput), which is one of the inputs to [schedule the model-monitoring job](https://sagemaker.readthedocs.io/en/stable/api/inference/model_monitor.html#sagemaker.model_monitor.model_monitoring.ModelMonitor.create_monitoring_schedule).

**Describe which SageMaker services are involved**
1. Endpoints
2. Model Quality Monitor

**Describe what other services (other than SageMaker) are involved***
S3: to store the captured data and ground truths.
CloudWatch: to capture the metrics and for alarms.

**Describe which dataset could be used. Provide its location in s3://sagemaker-sample-files or another source.**
1. MNIST dataset: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html
3. Adult dataset: https://archive.ics.uci.edu/ml/datasets/adult
4. Shelter Animal Outcome: https://www.kaggle.com/c/shelter-animal-outcomes/data

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the EndpointInput and ModelMonitor.create_monitoring_schedule documentation linked in the issue, then examine the available SageMaker example notebooks. Build an example around a multi-class model using one of the suggested datasets, showing prediction and label JSON paths, scheduled model-quality monitoring, S3 data, and CloudWatch metrics or alarms. Done means the notebook demonstrates these steps end to end.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, scikit-learn, tensorflow
Domain
cloud, machine-learning, observability
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.