allenai / allenai/scibert

Unused Weights When Loading Model

Aperta
#108 0 commenti 2 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
1.7k
Fork
232
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

I am using the newest version of Transformers (i.e. version 3.0) and try to load a Scibert model using the following code

`scibert_tokenizer = AutoTokenizer.from_pretrained('allenai/scibert_scivocab_uncased')

scibert_model = BertForMaskedLM.from_pretrained('allenai/scibert_scivocab_uncased')
`

When I do this I get the following message indicating unused weights:

`Some weights of the model checkpoint at allenai/scibert_scivocab_uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']

This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).`

I do not get a similar error when using an earlier version of Transformers (i.e. 2.something).

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.