explosion / explosion/spaCy

Memory usage of `debug-data` with a huge training set

Open
#4,748 5 comments 0 reactions 0 assignees View on GitHub
enhancement feat / cli perf / memory
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

Hi, I'm using Spacy 2.2.2 to train new tagger and parser models for the Italian language.
My training data set is quite big (about 2.3 GB for the train and 580 MB for the dev) and is saved in two JSONL files.
I'm experiencing an unexpected memory usage when running the `debug-data` command: memory usage starts low and then grows up to consuming my 32GB of RAM as well as the whole swap (about the same size).
Before upgrading my RAM to 128 GB (which I suspect might be useless), I'm interested in your opinion about:
* hints about data set structure: for instance, comments in issue #4700 suggested to reduce the sentence length on average, but I've no clue about what values might be optimal; is there any rule of thumb to properly dimension the data set?
* possible optimizations to the source code to reduce memory footprint (for instance by improving the lazy loading of the data set); I'm willing to contribute to Spacy if anyone would kindly point me to the problematic parts (if any, of course)

## Info about spaCy

* **spaCy version:** 2.2.2
* **Platform:** Linux-4.4.0-112-generic-x86_64-with-debian-stretch-sid
* **Python version:** 3.7.4

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.