cdpierse / cdpierse/transformers-interpret

Support for Longformer

Open
#130 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.4k
Forks
99
PR merge metrics
No merged PRs in 30d

Description

I am trying to get word attributions running for the Longformer model.

```
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
from transformers_interpret import QuestionAnsweringExplainer

question = "How many programming languages does BLOOM support?"
context = "BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages."

tokenizer = AutoTokenizer.from_pretrained("valhalla/longformer-base-4096-finetuned-squadv1")
model = AutoModelForQuestionAnswering.from_pretrained("valhalla/longformer-base-4096-finetuned-squadv1")

qa_explainer = QuestionAnsweringExplainer(
model,
tokenizer,
)

word_attributions = qa_explainer(
question,
context,
)

print(word_attributions)
print(qa_explainer.predicted_answer)
qa_explainer.visualize("bert_qa_viz.html")
```

With this code I get the following error.
```AssertionError: There should be exactly three separator tokens: 2 in every sample for questions answering. You might also consider to set `global_attention_mask` manually in the forward function to avoid this error.```

Is this expected? How do I get the word attributions for the Longformer model (if I can)?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.