aws / aws/amazon-sagemaker-clarify

Multi-categorical confusion matrix calculation for labels not presented in predicted_labels

Open
#138 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
75
Forks
41
PR merge metrics
No merged PRs in 30d

Description

Feedback from Bilal from a PR review: https://github.com/aws/amazon-sagemaker-clarify/pull/136#discussion_r1124552393

```
How are we supposed to handle cases when a predicted label (in this case 2) is not present in the observed labels (in this case [1])? Some options are:

We limit the confusion matrix CM to labels are present in both observed (label_series) and predicted labels (predicted_label_series). This is what sklearn does.
CM contains labels from the union of observed and predicted labels.
CM contains labels from observed labels only. If a predicted label is not found in observed labels, we raise an error saying something like "Unknown label 2".
I think we should pick option 3 since it assumes that observed labels provide us a complete list of all the possible labels. Option 1 could be problematic because it will drop some valid observed labels in case they are not found in predicted labels.

If we opt for 3, we should raise an error in this line.
```

Need to figure out if we want to handle this from analyzer side or library.

Contributor guide

Open the contributing guide

Research direction

Start with the PR 136 review discussion and trace where the multi-categorical confusion matrix is calculated, including whether the decision belongs in the analyzer or library. Define the policy for predicted labels absent from observed labels, then add coverage for that case and verify the selected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.