HumanSignal / HumanSignal/label-studio

Persistency when running the ML backend

Open
#590 2 comments 0 reactions 1 assignee Claimed by @makseq View on GitHub
feature ML SDK
Dominant language
TypeScript
Stars
28.3k
Forks
3.7k
Avg merge
14h
Merged PRs (30d)
15

Description

I am using LS to label audio data. Is use a local file dir, and access the audio files in the `fit` method as (sample code)
```
for k in completions:
print("Choice: {}".format(k['completions'][-1]['result'][0]['value']['choices']))
print("for file {}".format(k['task_path']))
y, sr = librosa.load(k['task_path'])
print(y.shape, sr)
```
I noticed that the script gets re-run all over again when starting the training. Loading the audio data is time-expensive, as is creating/loading a model. Is there a way to enable some kind of persistency?

I think of holding the audio files in RAM (I can provide enough RAM if necessary), for example in a `numpy` array. To this I would dynamically add any newly loaded audio file. I'd then check, during the next training whether I have previously loaded the file (via a dictionary, or something else).

A similar thing would be convenient for model handling. Instead of reloading, keep it in memory.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.