explosion / explosion/spaCy

Models are not deterministic / reproducible on GPU

Open
#6,490 4 comments 0 reactions 0 assignees View on GitHub
bug feat / ner gpu reproducibility
Dominant language
Python
Stars
33.9k
Forks
4.7k
Avg merge
3m
Merged PRs (30d)
1

Description

## How to reproduce the behaviour

I cannot reproduce the same results when training a NER model using GPU in Google Colab.
When running the same code with CPU it seems to work.
However, when enabling GPU with prefer_gpu() the reproduction is not working.

`
# Example code

def train_blank_ner_model(language_id, train_X, entity_types, epochs, random_state, dropout, minibatch_size, losses_display_frequency_in_epochs):
fix_random_seed(random_state)

nlp = spacy.blank(language_id)

assert len(nlp.pipe_names) == 0, f"Pipeline of blank model '{language_id}' is not empty."

ner = nlp.create_pipe('ner')

nlp.add_pipe(ner)

for entity_type in entity_types:
ner.add_label(entity_type)

optimizer = nlp.begin_training()

for epoch in tqdm(range(1, epochs + 1)):
train_X = shuffle(train_X, random_state=random_state)

losses = {}

batches = minibatch(train_X, size=compounding(*minibatch_size))

for batch in tqdm(batches, leave=False):
texts, annotations = zip(*batch)

nlp.update(texts, annotations, sgd=optimizer, drop=dropout, losses=losses)

if epoch % losses_display_frequency_in_epochs == 0:
print(f"Epoch {epoch}, Loss: {losses['ner']}")

print(f"Training completed with loss: {losses['ner']}")

return nlp

import spacy

print(f"GPU Initialization: {spacy.prefer_gpu()}")

nlp = train_blank_ner_model(language_id='de',
train_X=X_train,
entity_types=ner_entity_types,
epochs=3,
random_state=42,
dropout=0.4,
minibatch_size=(0.4, 0.4, 1.0),
losses_display_frequency_in_epochs=5)
`

## Your Environment

* Operating System: Google Colab
* Python Version Used: Python 3.8
* spaCy Version Used: spacy[cuda101]==2.3.4
* Environment Information: Google Colab

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.