dotnet / dotnet/machinelearning

[ImageClassification Transfer Learning] Need simplified API for obtaining the List of predicted labels with their related scoring

Open
#4,083 0 comments 1 reaction 1 assignee Claimed by @codemzs View on GitHub
area-Vision enhancement Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

In the current implementation of ImageClassification Transfer Learning (but this also happens when scoring regular TensorFlow models) the list of predicted labels (usually categorical data of type string/text) with their related scoring (probability) is not straightforward to get.

We need a very simple and straightforward way of getting the best predicted values (such as categorical data) with their related probability.

Currently, in a typical ImagePrediction class you might have (such as in the sample link):

https://github.com/dotnet/machinelearning/blob/bbb6b1560dbd67640cf9c4b41640e26769ee664d/docs/samples/Microsoft.ML.Samples/Dynamic/ImageClassification/InceptionV3TransferLearning.cs#L101

```
public class ImagePrediction
{
[ColumnName("Score")]
public float[] Score;

[ColumnName("PredictedLabel")]
public Int64 PredictedLabel;
}
```

1. First of all, the predicted label you get is a number (Int64) which is an **index**, not the value itself such as a categorical value. How is the user supposed to easily find out the text/categorical predicted label? - Sure, we might be able to find out through the schema API, but that is not straightforward.

2. With the `PredictedLabel` field we're only getting **one** predicted label (so far, its index), but the user would probably like to get an array with the "best" or even "all" predicted labels plus their correlated score/probability.

3. The float[] array `Score` is precisely getting the probabilities for all the labels. But, how is the user supposed to easily be able to correlate those scores with their related labels (categorical values, for instance)?

**ACTIONS MOVING FORWARD:**

1. As initial step, the sample code above (or new sample) should show on the console output the list of all labels in their original form (categorical/text), or at least the best three predicted labels and their related score/probability. This should be done in the sample with the current API capabilities even if it is not straightforward to code for a user.

2. Moving forward, since our goal is to create a simplified and high-level API for ImageClassification Transfer Learning, we should provide a way where such information is straightforward and **directly provided by the "Prediction class"**.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.