HumanSignal / HumanSignal/label-studio-ml-backend

Webhook not deleted when ML backend deleted

Open
#268 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
490
Avg merge
1d 14h
Merged PRs (30d)
3

Description

Hello! I have encountered a bug when linking custom ML backends with Label Studio. When explicitly deleting an ML backend, the corresponding webhook for the backend is not also deleted.

For example, if I set up a custom YoloV5 ML backend at 10.40.52.3:9090 then a webhook is created for 10.40.52.3:9090.

```
# Start the Label Studio ML Backend with the YoloV5 backend
label-studio-ml init yolov5 --script ./yolov5_autolabeling_backend.py
label-studio-ml start ./yolov5 -p 9090 --log-level DEBUG
```

![ls_1](https://github.com/heartexlabs/label-studio-ml-backend/assets/33724687/01a4ac45-c4db-4647-b784-aa5a2d6a4652)

![ls_2](https://github.com/heartexlabs/label-studio-ml-backend/assets/33724687/23405f9a-bcdd-486a-b027-280f894f0a2d)

But if I delete that custom YoloV5 ML backend at 10.40.52.3:9090 from the UI, the webhook persists.

```
CTRL+C for the previous label-studio-ml start process
```

![ls_3](https://github.com/heartexlabs/label-studio-ml-backend/assets/33724687/eb507518-c241-4be3-acfa-b1e1ac190ccc)

![ls_4](https://github.com/heartexlabs/label-studio-ml-backend/assets/33724687/891c63ec-789b-4d97-a60a-3e480edda454)

If, at some point in the future, I attempt to link a new YoloV5 ML backend at 10.40.52.3:**9091** from the UI, there will be two webhooks.

![ls_5](https://github.com/heartexlabs/label-studio-ml-backend/assets/33724687/a5083ee5-3ed5-4442-a505-71b6dea30872)

![ls_6](https://github.com/heartexlabs/label-studio-ml-backend/assets/33724687/0cd37422-8d1d-416b-8d0d-acdd98200f4c)

When the specified events handled by the webhooks occur (for example, an ANNOTATION_UPDATED event), Label Studio attempts to send the notification to both URLs. However, if one of the URLs doesn't exist - like how there is no longer a YoloV5 ML backend at 10.40.52.3:9090, the process will fail silently. Not only will the notification not reach the non-existent backend, _it won't reach any of the backends._

See the attached logs from the 10.40.52.3:**9091** label studio backend and the label studio container, where this exact situation played out. I attempted to update an annotation in a project with its custom YoloV5 ML backend at 10.40.52.3:9091 - and this backend never received the ANNOTATION_UPDATED event. The container shows that attempting to send notification of the ANNOTATION_UPDATED event to 10.40.52.3:9090 failed - and that finally led me to look at the webhooks page in the UI.

[ls_backend_logs.txt](https://github.com/heartexlabs/label-studio-ml-backend/files/11500151/ls_backend_logs.txt)

[ls_container_docker_log.txt](https://github.com/heartexlabs/label-studio-ml-backend/files/11500051/ls_container_docker_log.txt)

A few potential solutions:
* Delete webhooks after explicit deletion of their corresponding models
* Notify users with a pop-up upon model deletion that they may have persistent webhooks

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.