argosopentech / argosopentech/argos-train

Is there a way to reduce RAM consumption during training?

Open
#45 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
158
Forks
29
PR merge metrics
No merged PRs in 30d

Description

Training e.g. fr->de with the complete data-index.json soon comes to the point that my RAM memory (32 GB) is completely used up and the process gets stuck.
Is there maybe one of the parameters in config.yml suitable for preventing/mitigating this??
Or do you have any other idea (besides reducing the data set)??

Very grateful for any hint!
kr, Andi

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading config.yml and tracing how the complete data-index.json is loaded during fr->de training. Determine whether an existing parameter can reduce peak RAM use or whether a design change is needed; done means documenting or implementing a validated way to avoid the reported 32 GB memory exhaustion without reducing the dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.