HumanSignal / HumanSignal/label-studio

[Feature] Filter by Individual Prediction Score / Metadata

Open
#9,792 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
28.3k
Forks
3.7k
Avg merge
14h
Merged PRs (30d)
15

Description

**Is your feature request related to a problem? Please describe.**
I have trained a model and i would like to know if it performs well and on which datarow it is performing well or not.

**Describe the solution you'd like**
I would like to be able to filter **individual** prediction results in labelstudio by their score.

More generally :
- I would like to be able to add one or more metrics to a prediction result (for ex : Score, IoU)
- I would like to be able to add one or more attributes to a prediction result (for ex : Status : "TruePositive")

Image

In this list, there would be two new possibilities :
- pred. result - IoU (from metrics dict) -> filtrable by comparison
- pred. result - Status (from attribute dict) -> filtrable by content
- pred.result - Set (from attribute dict -> filterable by content (exemple train/validation/test)

I would like to be able to filter theses prediction fields directly in the data manager.

**Describe alternatives you've considered**
I can do this by hand but it's very time consuming. I can use another tool like Voxel51, but because my data are stored in a "cloud like storage", i cannot use their base version.

**Impact Analysis**
In order to implement this, we'd need to modify the schema of a result, to contain theses field :
Image

We add two new entries :
- metrics : dict[str,int|float]
- attributes: dict[str,str]

They will be stored as json/dict in 2 separates columns in the prediction result database as a list of dictionary aligned on the result column (same number of values).

Image

In order to aware the data manager of the existence of attributes and metrics. We may add a kind of ontology here :

Image

In the ontology, we can add attributes and metrics.

This ontology would be necessary for two reason :
- aware the datamanager of the existence of the fields to allow filtering on them
- filter metrics and attributes during the prediction ingestion in order to only authorize medatadata/attributes that are declared.

It would be stored in a prediction_ontology database (one project == one ontology).

Contributor guide

Open the contributing guide

Research direction

Start by reading the data manager, prediction-result schema, prediction ingestion, and project ontology paths to understand how result fields are stored and exposed for filtering. Done means metrics and attributes can be declared, ingested, stored per prediction result, and filtered in the data manager, with coverage for the supported comparisons and content filters.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data, databases, full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.