dotnet / dotnet/machinelearning-modelbuilder

Consider showing balanced accuracy instead of accuracy + additional accuracy metrics

Open
#1,996 2 comments 0 reactions 0 assignees View on GitHub
Feature request Reported by: Customer Stale
Dominant language
Dockerfile
Stars
285
Forks
66
PR merge metrics
No merged PRs in 30d

Description

Currently, Model builder shows accuracy for the user because it is intuitive to understand.
However, in cases on imbalanced datasets, accuracy can be misleading. I believe real-world datasets are rarely perfectly balanced and in this cases it is no longer intuitive to understand what accuracy means.

For examples, I might have 10% false and 90% true in my dataset. If Model would predict everything as true I still get 90% accuracy.

The issues may even be worse in near-balanced datasets, for example, 60/40% because user might not even think about this possibility.

Even if ML.NET does not have average / balanced accuracy score, it can be calculated from other metrics.

Alternatives for, or additions to, balanced accuracy:
- show the distribution of test data across classes.
- show confusion matrix
- show null accuracy

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.