HumanSignal / HumanSignal/label-studio

Custom validator for annotations

Open
#8,065 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
TypeScript
Stars
28.3k
Forks
3.7k
Avg merge
14h
Merged PRs (30d)
15

Description

**Is your feature request related to a problem? Please describe.**
For our OCR-labeling we have evolved to a quite a complex structure with some implicit assumptions that are not integrated into the label-format itself. (E.g.: Boxes with label-type "Currency" need to always have a relation to a Box with label-type "Price", if there is an box with label-type "street number", we expect a box with label-type "street name", some boxes are not allowed to overlap, some boxes are supposed to have a box with a certain label-type that they overlap with, ...)

When switching between different annotators over time, they will naturally make some mistakes regarding that format that then need to be corrected.

**Describe the solution you'd like**
I would like to provide a custom validator via an API that works similar to webhooks, but can also return a list of errors, e.g. in the following format:
```
{
"global-errors": [
"there is always a box with label x required", "there are too many boxes with label y"
],
"per-result-errors": {
"tRvQ1XBXzQ": ["Boxes with label z expects to have an relation set."]
}
}
```
This would then be integrated to check the annotation upon saving and immediately displays them in the UI (global errors in the UI, results with errors on highlight. Annotators that are in the labeling interface won't just forward to the next task, but stay on the same task, if there are errors present.

**Describe alternatives you've considered**
* Just using webhooks: We have tried using webhooks that analyse updated tasks and send the respective annotator a slack-message with the errors. This works, but is cumbersome, as the annotators have to switch program and figure out themselves, which results produced the error, leading to cognitive overhead.
* Extending the labeling format with constraints: This could work, however I suppose that with a custom validator implemented, we would become more creative with ways of validation, so it is hard to come up with a structure that covers all - not yet thought of - ways to validate annotations. So as a developer, I would prefer to just validate in python code instead of predefined fields.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.