HumanSignal / HumanSignal/label-studio-ml-backend
Incorrect Label Assignment for Multiple Objects in Label Studio ML Backend
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 490
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 3
Description
**Description:**
I discovered the problem when using Label Studio ML Backend to label a video with multiple objects to track. This issue occurs consistently with all videos that contain multiple objects, regardless of their complexity or duration. All objects in a task receive the same label in the user interface, regardless of the distinct labels assigned by the model.
**Key Issue:**
The core problem is that when updating annotations via the model's response, the Label Studio interface is expected to display distinct labels for each object. However, it ends up displaying identical labels for all objects, even if the labels are supposed to be different. This happens both when creating new annotations and updating existing ones.
**Request:**
- Ensure that when multiple objects are present in the predictions, all of them are correctly displayed in the UI with their respective labels.
**Steps to Reproduce:**
1. Start by using a label config template for video object tracking: [https://labelstud.io/templates/video_object_detector](https://labelstud.io/templates/video_object_detector).
2. Configure Label Studio ML Backend and return the following dummy prediction. A 'dummy prediction' in this context refers to a sample output generated by the ML backend to simulate how predictions would look for multiple objects being tracked. This example demonstrates how the ML backend is expected to return prediction data for multiple tracked objects. The goal is to ensure that each object is uniquely labeled, which aligns with the intended workflow of providing distinct labels for different objects in the Label Studio UI:
```python
def predict(...):
import json
prediction = json.dumps({
"model_version": None,
"score": 0.0,
"result": [
{
"value": {
"framesCount": 43,
"duration": 4.3,
"sequence": [
{
"frame": 1,
"x": 51.42,
"y": 50.35,
"width": 3.41,
"height": 1.39,
"enabled": True,
"rotation": 0,
"time": 0.0
},
{
"frame": 2,
"x": 50.43,
"y": 50.35,
"width": 3.41,
"height": 1.39,
"enabled": True,
"rotation": 0,
"time": 0.1
}
],
"labels": [
"Woman"
]
},
"from_name": "box",
"to_name": "video",
"type": "videorectangle",
"origin": "manual",
"id": "mGjkBhvZpv"
},
{
"value": {
"framesCount": 43,
"duration": 4.3,
"sequence": [
{
"frame": 1,
"x": 0.0,
"y": 50.35,
"width": 78.98,
"height": 49.31,
"enabled": True,
"rotation": 0,
"time": 0.0
},
{
"frame": 2,
"x": 0.0,
"y": 50.87,
"width": 78.55,
"height": 48.78,
"enabled": True,
"rotation": 0,
"time": 0.1
}
],
"labels": [
"Man"
]
},
"from_name": "box",
"to_name": "video",
"type": "videorectangle",
"origin": "manual",
"id": "dssMyWHsv_"
}
]
})
return ModelResponse(predictions=[json.loads(prediction)])
```
3. Observe that all objects in the UI are incorrectly assigned the label of the first object from the predictions, instead of displaying their respective unique labels as provided by the model.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with the video object tracking template and the dummy prediction shown in the report. Trace how the ML backend handles multiple prediction results and verify that the UI preserves each result's label; done means objects display their respective labels when creating or updating annotations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100