Make it possible to merge Vocab/StringStore instances
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## Feature description
Here is the background: in order to use a spacy document it is necessary to have the correct Vocab/StringStore. But when documents are created/processed in a distributed/multiprocessing way, different subsets of documents will get processed, linked to the vocab of the process's
vocab (in nlp). In order to save a processed document efficiently, one uses "to_disk" to save it without the vocab.
When resuming processing or just when one wants to load the document later, a vocab is needed that has all the entries from all the parallel processes combined in order to ensure the document can be deserialised.
## Could the feature be a custom component
I do not think so.
Contributor guide
Assessment
This issue has not been assessed yet.