argosopentech / argosopentech/argos-train
Is there a way to reduce RAM consumption during training?
- Dominant language
- Python
- Stars
- 158
- Forks
- 29
- PR merge metrics
- No merged PRs in 30d
Description
Training e.g. fr->de with the complete data-index.json soon comes to the point that my RAM memory (32 GB) is completely used up and the process gets stuck.
Is there maybe one of the parameters in config.yml suitable for preventing/mitigating this??
Or do you have any other idea (besides reducing the data set)??
Very grateful for any hint!
kr, Andi
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading config.yml and tracing how the complete data-index.json is loaded during fr->de training. Determine whether an existing parameter can reduce peak RAM use or whether a design change is needed; done means documenting or implementing a validated way to avoid the reported 32 GB memory exhaustion without reducing the dataset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100