HumanSignal / HumanSignal/label-studio
Potential memory leak in prediction retrieval
- Dominant language
- TypeScript
- Stars
- 28.3k
- Forks
- 3.7k
- Avg merge
- 14h
- Merged PRs (30d)
- 15
Description
Hey all, just hoping if anybody can provide some insight to memory issues I’ve been encountering with our Label Studio deployment.
We are running version 1.1.0 in a manner similar to your docker deployment with containers for app, postgres, nginx, ml_server and redis.
Problems occur when attempting to retrieve predictions for the whole dataset (40k tasks, 10k annotations) through the UI (by toggling select all and retrieve predictions from the dropdown).
App container memory usage after build is < 300MiB and remains stable when labelling. When attempting to retrieve predictions memory usage climbs to around 1.4GiB and does not dissipate when the task completes. If I then attempt to retrieve predictions again, memory usage climbs further and usually leads to the app container exiting with code 137 (out of memory) or sometimes just crashing the host (Google Cloud e2-medium virtual machine with 3.8GiB usable memory running Ubuntu 20.04.2).
The UI recommends using an API post call to /api/predictions to fulfil this function however without the ability to batch tasks on this endpoint its unfeasible time wise for 40k tasks.
Let me know if there are further details I can provide.
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.