cdpierse / cdpierse/transformers-interpret
Support for Longformer
- Dominant language
- Jupyter Notebook
- Stars
- 1.4k
- Forks
- 99
- PR merge metrics
- No merged PRs in 30d
Description
I am trying to get word attributions running for the Longformer model.
```
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
from transformers_interpret import QuestionAnsweringExplainer
question = "How many programming languages does BLOOM support?"
context = "BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages."
tokenizer = AutoTokenizer.from_pretrained("valhalla/longformer-base-4096-finetuned-squadv1")
model = AutoModelForQuestionAnswering.from_pretrained("valhalla/longformer-base-4096-finetuned-squadv1")
qa_explainer = QuestionAnsweringExplainer(
model,
tokenizer,
)
word_attributions = qa_explainer(
question,
context,
)
print(word_attributions)
print(qa_explainer.predicted_answer)
qa_explainer.visualize("bert_qa_viz.html")
```
With this code I get the following error.
```AssertionError: There should be exactly three separator tokens: 2 in every sample for questions answering. You might also consider to set `global_attention_mask` manually in the forward function to avoid this error.```
Is this expected? How do I get the word attributions for the Longformer model (if I can)?
Contributor guide
Assessment
This issue has not been assessed yet.