dotnet / dotnet/machinelearning

Add baseline metrics to trainer results

Open
#6,796 0 comments 0 reactions 0 assignees View on GitHub
area-Core enhancement untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Datasets are almost never perfectly balanced. That means that an impressive metric, let's say 80% binary classification accuracy, could be worse than selecting always the most common category. Therefore, I would like to automatically calculate the baseline metrics, for example:
- null accuracy
- distribution of items according to categories
- baseline metric for ranking of a random order

I think ideally, I would be able to read the baseline metrics as easily as the training data metrics. I think it should be also promoted as during the currenty AI summer I see many developers quickly build AI-enabled apps without any or much validation, i.e. whatever GPT models says is enough without validation or even real prompt engineering with comparison to ground truth; whatever Model builder says is the final validation and so on.

I think this would be in line with promoting fairness values. Equally incorrect results for everyone is not fair. Model Builder could probably even show graphically how much better model performs on test dataset than random or null accuracy.

First I think it would be important to record what is the ideal way to evaluate metrics. For example, comparison to null accuracy would probably be good for binary classification but not others. For ranking, there seems to be a few options. Implementation is probably not terribly difficult.

1. Decide baseline metrics for each ML task
2. Decide programming interface and where to calculate the metrics
3. Implementation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.