AlibabaResearch / AlibabaResearch/AdvancedLiterateMachinery

GeoLayoutLM: Warning about mismatched tokenizer class: BrosTokenizer vs BertTokenizer

Aperta
#211 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C++
Stelle
1.8k
Fork
195
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Hi, I encountered this warning when running the code:

The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'BertTokenizer'.
The class this function is called from is 'BrosTokenizer'.

However, in the code I only see this line:

TOKENIZER = BertTokenizer.from_pretrained("bert-large-uncased", do_lower_case=True)

I couldn't find any BrosTokenizer in the codebase. Is this warning expected, or could there be a mismatch somewhere in how the tokenizer is being used or loaded?

Thanks!

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.