HumanSignal / HumanSignal/label-studio
Persistency when running the ML backend
- Dominant language
- TypeScript
- Stars
- 28.3k
- Forks
- 3.7k
- Avg merge
- 14h
- Merged PRs (30d)
- 15
Description
I am using LS to label audio data. Is use a local file dir, and access the audio files in the `fit` method as (sample code)
```
for k in completions:
print("Choice: {}".format(k['completions'][-1]['result'][0]['value']['choices']))
print("for file {}".format(k['task_path']))
y, sr = librosa.load(k['task_path'])
print(y.shape, sr)
```
I noticed that the script gets re-run all over again when starting the training. Loading the audio data is time-expensive, as is creating/loading a model. Is there a way to enable some kind of persistency?
I think of holding the audio files in RAM (I can provide enough RAM if necessary), for example in a `numpy` array. To this I would dynamically add any newly loaded audio file. I'd then check, during the next training whether I have previously loaded the file (via a dictionary, or something else).
A similar thing would be convenient for model handling. Instead of reloading, keep it in memory.
Contributor guide
Assessment
This issue has not been assessed yet.