Models are not deterministic / reproducible on GPU
- Dominant language
- Python
- Stars
- 33.9k
- Forks
- 4.7k
- Avg merge
- 3m
- Merged PRs (30d)
- 1
Description
## How to reproduce the behaviour
I cannot reproduce the same results when training a NER model using GPU in Google Colab.
When running the same code with CPU it seems to work.
However, when enabling GPU with prefer_gpu() the reproduction is not working.
`
# Example code
def train_blank_ner_model(language_id, train_X, entity_types, epochs, random_state, dropout, minibatch_size, losses_display_frequency_in_epochs):
fix_random_seed(random_state)
nlp = spacy.blank(language_id)
assert len(nlp.pipe_names) == 0, f"Pipeline of blank model '{language_id}' is not empty."
ner = nlp.create_pipe('ner')
nlp.add_pipe(ner)
for entity_type in entity_types:
ner.add_label(entity_type)
optimizer = nlp.begin_training()
for epoch in tqdm(range(1, epochs + 1)):
train_X = shuffle(train_X, random_state=random_state)
losses = {}
batches = minibatch(train_X, size=compounding(*minibatch_size))
for batch in tqdm(batches, leave=False):
texts, annotations = zip(*batch)
nlp.update(texts, annotations, sgd=optimizer, drop=dropout, losses=losses)
if epoch % losses_display_frequency_in_epochs == 0:
print(f"Epoch {epoch}, Loss: {losses['ner']}")
print(f"Training completed with loss: {losses['ner']}")
return nlp
import spacy
print(f"GPU Initialization: {spacy.prefer_gpu()}")
nlp = train_blank_ner_model(language_id='de',
train_X=X_train,
entity_types=ner_entity_types,
epochs=3,
random_state=42,
dropout=0.4,
minibatch_size=(0.4, 0.4, 1.0),
losses_display_frequency_in_epochs=5)
`
## Your Environment
* Operating System: Google Colab
* Python Version Used: Python 3.8
* spaCy Version Used: spacy[cuda101]==2.3.4
* Environment Information: Google Colab
Contributor guide
Assessment
This issue has not been assessed yet.