HumanSignal / HumanSignal/label-studio-ml-backend

Incorrect Label Assignment for Multiple Objects in Label Studio ML Backend

Open
#664 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
490
Avg merge
1d 14h
Merged PRs (30d)
3

Description

**Description:**

I discovered the problem when using Label Studio ML Backend to label a video with multiple objects to track. This issue occurs consistently with all videos that contain multiple objects, regardless of their complexity or duration. All objects in a task receive the same label in the user interface, regardless of the distinct labels assigned by the model.

**Key Issue:**
The core problem is that when updating annotations via the model's response, the Label Studio interface is expected to display distinct labels for each object. However, it ends up displaying identical labels for all objects, even if the labels are supposed to be different. This happens both when creating new annotations and updating existing ones.

**Request:**

- Ensure that when multiple objects are present in the predictions, all of them are correctly displayed in the UI with their respective labels.

**Steps to Reproduce:**

1. Start by using a label config template for video object tracking: [https://labelstud.io/templates/video_object_detector](https://labelstud.io/templates/video_object_detector).

2. Configure Label Studio ML Backend and return the following dummy prediction. A 'dummy prediction' in this context refers to a sample output generated by the ML backend to simulate how predictions would look for multiple objects being tracked. This example demonstrates how the ML backend is expected to return prediction data for multiple tracked objects. The goal is to ensure that each object is uniquely labeled, which aligns with the intended workflow of providing distinct labels for different objects in the Label Studio UI:

```python
def predict(...):
import json
prediction = json.dumps({
"model_version": None,
"score": 0.0,
"result": [
{
"value": {
"framesCount": 43,
"duration": 4.3,
"sequence": [
{
"frame": 1,
"x": 51.42,
"y": 50.35,
"width": 3.41,
"height": 1.39,
"enabled": True,
"rotation": 0,
"time": 0.0
},
{
"frame": 2,
"x": 50.43,
"y": 50.35,
"width": 3.41,
"height": 1.39,
"enabled": True,
"rotation": 0,
"time": 0.1
}
],
"labels": [
"Woman"
]
},
"from_name": "box",
"to_name": "video",
"type": "videorectangle",
"origin": "manual",
"id": "mGjkBhvZpv"
},
{
"value": {
"framesCount": 43,
"duration": 4.3,
"sequence": [
{
"frame": 1,
"x": 0.0,
"y": 50.35,
"width": 78.98,
"height": 49.31,
"enabled": True,
"rotation": 0,
"time": 0.0
},
{
"frame": 2,
"x": 0.0,
"y": 50.87,
"width": 78.55,
"height": 48.78,
"enabled": True,
"rotation": 0,
"time": 0.1
}
],
"labels": [
"Man"
]
},
"from_name": "box",
"to_name": "video",
"type": "videorectangle",
"origin": "manual",
"id": "dssMyWHsv_"
}
]
})
return ModelResponse(predictions=[json.loads(prediction)])
```

3. Observe that all objects in the UI are incorrectly assigned the label of the first object from the predictions, instead of displaying their respective unique labels as provided by the model.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with the video object tracking template and the dummy prediction shown in the report. Trace how the ML backend handles multiple prediction results and verify that the UI preserves each result's label; done means objects display their respective labels when creating or updating annotations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.