Memory usage of `debug-data` with a huge training set
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
Hi, I'm using Spacy 2.2.2 to train new tagger and parser models for the Italian language.
My training data set is quite big (about 2.3 GB for the train and 580 MB for the dev) and is saved in two JSONL files.
I'm experiencing an unexpected memory usage when running the `debug-data` command: memory usage starts low and then grows up to consuming my 32GB of RAM as well as the whole swap (about the same size).
Before upgrading my RAM to 128 GB (which I suspect might be useless), I'm interested in your opinion about:
* hints about data set structure: for instance, comments in issue #4700 suggested to reduce the sentence length on average, but I've no clue about what values might be optimal; is there any rule of thumb to properly dimension the data set?
* possible optimizations to the source code to reduce memory footprint (for instance by improving the lazy loading of the data set); I'm willing to contribute to Spacy if anyone would kindly point me to the problematic parts (if any, of course)
## Info about spaCy
* **spaCy version:** 2.2.2
* **Platform:** Linux-4.4.0-112-generic-x86_64-with-debian-stretch-sid
* **Python version:** 3.7.4
Contributor guide
Assessment
This issue has not been assessed yet.