explosion / explosion/spaCy

Make it possible to merge Vocab/StringStore instances

Open
#4,411 9 comments 0 reactions 0 assignees View on GitHub
enhancement feat / serialize scaling
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## Feature description

Here is the background: in order to use a spacy document it is necessary to have the correct Vocab/StringStore. But when documents are created/processed in a distributed/multiprocessing way, different subsets of documents will get processed, linked to the vocab of the process's
vocab (in nlp). In order to save a processed document efficiently, one uses "to_disk" to save it without the vocab.
When resuming processing or just when one wants to load the document later, a vocab is needed that has all the entries from all the parallel processes combined in order to ensure the document can be deserialised.

## Could the feature be a custom component

I do not think so.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.