explosion / explosion/spaCy

Spacy Language components initialization extremely slow during cli training due to corpus train size.

Open
#13,804 1 comment 0 reactions 0 assignees View on GitHub
feat / training feat / ux
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour

run cli train for a over 10g of training data (DocBin). it takes several hours to initialize. the train hasn't started.
My use case is transformer+ner. I have about 1000 entity labels with 2 million training docs. After 10 hours I had to abort.
## Your Environment

* Operating System: linux
* Python Version Used: 3.12
* spaCy Version Used: latest v4
* Environment Information:GCP A1-ULTRA 16 cores 170gb MEM , A100 80GB GPU
```
ython3 -m cProfile -o output.prof -m spacy train ./DONOT_MODIFY_en_core_web_custom_ent.cfg --paths.train ./train_spacy_4m_small --paths.dev ./test_spacy_4m --training.max_epochs 5 --gpu-id 0 --output ./custom_ent_4mtrain --verbose
[2025-04-21 18:52:36,710] [DEBUG] Config overrides from CLI: ['paths.train', 'paths.dev', 'training.max_epochs']
ℹ Saving to output directory: custom_ent_4mtrain
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2025-04-21 18:52:38,416] [INFO] Set up nlp object from config
[2025-04-21 18:52:38,457] [DEBUG] Loading corpus from path: test_spacy_4m
[2025-04-21 18:52:38,461] [DEBUG] Loading corpus from path: train_spacy_4m_small
[2025-04-21 18:52:38,461] [INFO] Pipeline: ['transformer', 'ner', 'doc_cleaner']
[2025-04-21 18:52:38,485] [DEBUG] Loading lookups from spacy-lookups-data: ['lexeme_norm']
[2025-04-21 18:52:38,507] [INFO] Added vocab lookups: lexeme_norm
[2025-04-21 18:52:38,507] [INFO] Created vocabulary
[2025-04-21 18:52:38,507] [INFO] Finished initializing nlp object
----- took 10 hours after which I had to abort it.
```
possible issue identified [here](https://github.com/explosion/spaCy/blob/acb44f8e73ef04b4b019637d5e72e6ad92508e73/spacy/language.py#L1455-L1464)
```
self.tokenizer.initialize(get_examples, nlp=self, **tok_settings) # type: ignore[union-attr]
for name, proc in self.pipeline:
if isinstance(proc, ty.InitializableComponent):
p_settings = I["components"].get(name, {})
if labels is not None and name in labels:
p_settings["labels"] = labels[name]
p_settings = validate_init_settings(
proc.initialize, p_settings, section="components", name=name
)
proc.initialize(get_examples, nlp=self, **p_settings)
pretrain_cfg = config.get("pretraining")
```
request to run multiprocess to read through the examples for initialize . example below to speed up json to DocBin conversion from several hours to few minutes using multiprocess.

```
import spacy
from spacy.tokens import DocBin
from google.cloud import storage
import json

def batch_objects(objects, batch_size):
for i in range(0, len(objects), batch_size):
yield objects[i:i + batch_size]

def process_batch(indx_mini_batch):
try:
nlp = spacy.blank("en")
db = DocBin()
for text, annotations in indx_mini_batch[1]:
if len(text) > 10 and "entities" in annotations.keys() and len(annotations["entities"]) > 0:

doc = nlp.make_doc(text)
ents = []
for start, end, label in annotations['entities']:
span = doc.char_span(start, end, label=label)
if span:
ents.append(span)
doc.ents = ents
db.add(doc)
db.to_disk(f"/root/train_spacy_500k_v2/{indx_mini_batch[0]}.spacy")
return True
except Exception as e:
raise ValueError(f"{e}")

def run_all_batches(TRAIN_DATA):
objects = TRAIN_DATA # High volume of objects
batch_size = 100 # Define the batch size

# Create batches
batches = [x for x in enumerate(list(batch_objects(objects, batch_size)))]
with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor:
results= executor.map(process_batch, batches)
for result in results:
print(f"Result: {result}")

# Run the main function
run_all_batches(TRAIN_DATA_SPL)

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.