DAMO-NLP-SG / DAMO-NLP-SG/ContProto

Running Error for bash run_generate_pseudo.sh de conll0

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8
Forks
2
PR merge metrics
No merged PRs in 30d

Description

I try to run this code, however it seems that generate_pseudo.py could not runn correctly, the error is about this part:
dataset = BERTNERDataset(self.args, json_path=json_path,
tokenizer=BertWordPieceTokenizer(vocab_path),
# tokenizer=BertWordPieceTokenizer(vocab_file=vocab_path),
max_length=self.args.bert_max_length,
pad_to_maxlen=False
)
# dataset = BERTNERDataset(self.args, json_path=json_path,
# tokenizer=Tokenizer.from_pretrained(self.args.bert_config_dir),
# max_length=self.args.bert_max_length,
# pad_to_maxlen=False
# )

I'd like to know which type of tokenizer did you use when you train the model based on xlmr-large, It seems that both Tokenizer and BertWordPieceTokenizer Could not run correctly. I also tried the XLMRobertaTokenizer from transformers but its output is not incompatible with the output required by the current code. The code needs the output with ids,type_ids and offsets.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.