`create_kb` step using an unfiltered dump runs out of memory
- Dominant language
- Python
- Stars
- 41
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Hello!
I am working to create a knowledge base using the latest (unfiltered) English wiki dumps. I've successfully followed the steps in [benchmarks/nel](https://github.com/explosion/projects/tree/v3/benchmarks/nel) up to `wikid_parse` to make a 20GB `en/wiki.sqlite3` file.
However when I run the next step `wikid_create_kb`, my machine runs out of memory in two places:
1. retrieving entities [here](https://github.com/explosion/wikid/blob/94af5c6d0ebc5e960872d0ef54df20e030072a30/scripts/create_kb.py#L34C53-L34C53) which i think i resolved by modifying `PRAGMA mmap_size` in the SQLite table.
2. computing description vectors for all the entities [here](https://github.com/explosion/wikid/blob/94af5c6d0ebc5e960872d0ef54df20e030072a30/scripts/create_kb.py#L47). What kind of machine did y'all get this to work on? My estimate says that 16GB of memory should be fine but this step is quickly crashing my computer.
```
Running command: env PYTHONPATH=scripts python ./scripts/create_kb.py en_core_web_lg en
Inferring entity embeddings: 1%|▎
```
Here is a spacy info dump:
```
spaCy version 3.7.2
Location /opt/conda/lib/python3.10/site-packages/spacy
Platform Linux-5.10.0-26-cloud-amd64-x86_64-with-glibc2.31
Python version 3.10.12
Pipelines en_core_web_lg (3.7.0)
```
Thank you!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.