HumanSignal / HumanSignal/label-studio
[Feature] Filter by Individual Prediction Score / Metadata
- Dominant language
- TypeScript
- Stars
- 28.3k
- Forks
- 3.7k
- Avg merge
- 14h
- Merged PRs (30d)
- 15
Description
**Is your feature request related to a problem? Please describe.**
I have trained a model and i would like to know if it performs well and on which datarow it is performing well or not.
**Describe the solution you'd like**
I would like to be able to filter **individual** prediction results in labelstudio by their score.
More generally :
- I would like to be able to add one or more metrics to a prediction result (for ex : Score, IoU)
- I would like to be able to add one or more attributes to a prediction result (for ex : Status : "TruePositive")
In this list, there would be two new possibilities :
- pred. result - IoU (from metrics dict) -> filtrable by comparison
- pred. result - Status (from attribute dict) -> filtrable by content
- pred.result - Set (from attribute dict -> filterable by content (exemple train/validation/test)
I would like to be able to filter theses prediction fields directly in the data manager.
**Describe alternatives you've considered**
I can do this by hand but it's very time consuming. I can use another tool like Voxel51, but because my data are stored in a "cloud like storage", i cannot use their base version.
**Impact Analysis**
In order to implement this, we'd need to modify the schema of a result, to contain theses field :
We add two new entries :
- metrics : dict[str,int|float]
- attributes: dict[str,str]
They will be stored as json/dict in 2 separates columns in the prediction result database as a list of dictionary aligned on the result column (same number of values).
In order to aware the data manager of the existence of attributes and metrics. We may add a kind of ontology here :
In the ontology, we can add attributes and metrics.
This ontology would be necessary for two reason :
- aware the datamanager of the existence of the fields to allow filtering on them
- filter metrics and attributes during the prediction ingestion in order to only authorize medatadata/attributes that are declared.
It would be stored in a prediction_ontology database (one project == one ontology).
Contributor guide
Research direction
Start by reading the data manager, prediction-result schema, prediction ingestion, and project ontology paths to understand how result fields are stored and exposed for filtering. Done means metrics and attributes can be declared, ingested, stored per prediction result, and filtered in the data manager, with coverage for the supported comparisons and content filters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- data, databases, full-stack
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100