explosion / explosion/wikid

`create_kb` step using an unfiltered dump runs out of memory

Open
#32 4 comments 0 reactions 0 assignees View on GitHub
usage
Dominant language
Python
Stars
41
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Hello!

I am working to create a knowledge base using the latest (unfiltered) English wiki dumps. I've successfully followed the steps in [benchmarks/nel](https://github.com/explosion/projects/tree/v3/benchmarks/nel) up to `wikid_parse` to make a 20GB `en/wiki.sqlite3` file.

However when I run the next step `wikid_create_kb`, my machine runs out of memory in two places:

1. retrieving entities [here](https://github.com/explosion/wikid/blob/94af5c6d0ebc5e960872d0ef54df20e030072a30/scripts/create_kb.py#L34C53-L34C53) which i think i resolved by modifying `PRAGMA mmap_size` in the SQLite table.

2. computing description vectors for all the entities [here](https://github.com/explosion/wikid/blob/94af5c6d0ebc5e960872d0ef54df20e030072a30/scripts/create_kb.py#L47). What kind of machine did y'all get this to work on? My estimate says that 16GB of memory should be fine but this step is quickly crashing my computer.
```
Running command: env PYTHONPATH=scripts python ./scripts/create_kb.py en_core_web_lg en
Inferring entity embeddings: 1%|▎
```

Here is a spacy info dump:
```
spaCy version 3.7.2
Location /opt/conda/lib/python3.10/site-packages/spacy
Platform Linux-5.10.0-26-cloud-amd64-x86_64-with-glibc2.31
Python version 3.10.12
Pipelines en_core_web_lg (3.7.0)
```

Thank you!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.