HumanSignal / HumanSignal/label-studio-ml-backend
Webhook not deleted when ML backend deleted
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 490
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 3
Description
Hello! I have encountered a bug when linking custom ML backends with Label Studio. When explicitly deleting an ML backend, the corresponding webhook for the backend is not also deleted.
For example, if I set up a custom YoloV5 ML backend at 10.40.52.3:9090 then a webhook is created for 10.40.52.3:9090.
```
# Start the Label Studio ML Backend with the YoloV5 backend
label-studio-ml init yolov5 --script ./yolov5_autolabeling_backend.py
label-studio-ml start ./yolov5 -p 9090 --log-level DEBUG
```


But if I delete that custom YoloV5 ML backend at 10.40.52.3:9090 from the UI, the webhook persists.
```
CTRL+C for the previous label-studio-ml start process
```


If, at some point in the future, I attempt to link a new YoloV5 ML backend at 10.40.52.3:**9091** from the UI, there will be two webhooks.


When the specified events handled by the webhooks occur (for example, an ANNOTATION_UPDATED event), Label Studio attempts to send the notification to both URLs. However, if one of the URLs doesn't exist - like how there is no longer a YoloV5 ML backend at 10.40.52.3:9090, the process will fail silently. Not only will the notification not reach the non-existent backend, _it won't reach any of the backends._
See the attached logs from the 10.40.52.3:**9091** label studio backend and the label studio container, where this exact situation played out. I attempted to update an annotation in a project with its custom YoloV5 ML backend at 10.40.52.3:9091 - and this backend never received the ANNOTATION_UPDATED event. The container shows that attempting to send notification of the ANNOTATION_UPDATED event to 10.40.52.3:9090 failed - and that finally led me to look at the webhooks page in the UI.
[ls_backend_logs.txt](https://github.com/heartexlabs/label-studio-ml-backend/files/11500151/ls_backend_logs.txt)
[ls_container_docker_log.txt](https://github.com/heartexlabs/label-studio-ml-backend/files/11500051/ls_container_docker_log.txt)
A few potential solutions:
* Delete webhooks after explicit deletion of their corresponding models
* Notify users with a pop-up upon model deletion that they may have persistent webhooks
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.