cdpierse / cdpierse/transformers-interpret

ZeroShotClassificationExplainer appears to be broken

Open
#125 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.4k
Forks
99
PR merge metrics
No merged PRs in 30d

Description

To reproduce the issue, simply start an empty environment (e.g. google colab) and run

```python
! pip install transformers-interpret
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from transformers_interpret import ZeroShotClassificationExplainer
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")
model = AutoModelForSequenceClassification.from_pretrained("facebook/bart-large-mnli")
zero_shot_explainer = ZeroShotClassificationExplainer(model, tokenizer)
word_attributions = zero_shot_explainer(
"Today apple released the new Macbook showing off a range of new features found in the proprietary silicon chip computer. ",
labels = ["finance", "technology", "sports"],)
```

Where, if not in a notebook, simply remove `! pip install transformers-interpret`.

The output is:

```python
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting transformers-interpret
Downloading transformers_interpret-0.9.6-py3-none-any.whl (45 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.6/45.6 KB 1.4 MB/s eta 0:00:00
Collecting captum>=0.3.1
Downloading captum-0.6.0-py3-none-any.whl (1.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 32.8 MB/s eta 0:00:00
Collecting ipython<8.0.0,>=7.31.1
Downloading ipython-7.34.0-py3-none-any.whl (793 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 793.8/793.8 KB 37.4 MB/s eta 0:00:00
Collecting transformers>=3.0.0
Downloading transformers-4.26.0-py3-none-any.whl (6.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.3/6.3 MB 69.6 MB/s eta 0:00:00
Requirement already satisfied: torch>=1.6 in /usr/local/lib/python3.8/dist-packages (from captum>=0.3.1->transformers-interpret) (1.13.1+cu116)
Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (from captum>=0.3.1->transformers-interpret) (1.21.6)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from captum>=0.3.1->transformers-interpret) (3.2.2)
Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.8/dist-packages (from ipython<8.0.0,>=7.31.1->transformers-interpret) (5.7.1)
Requirement already satisfied: pygments in /usr/local/lib/python3.8/dist-packages (from ipython<8.0.0,>=7.31.1->transformers-interpret) (2.6.1)
Requirement already satisfied: backcall in /usr/local/lib/python3.8/dist-packages (from ipython<8.0.0,>=7.31.1->transformers-interpret) (0.2.0)
Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.8/dist-packages (from ipython<8.0.0,>=7.31.1->transformers-interpret) (4.8.0)
Collecting jedi>=0.16
Downloading jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 51.4 MB/s eta 0:00:00
Collecting matplotlib-inline
Downloading matplotlib_inline-0.1.6-py3-none-any.whl (9.4 kB)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from ipython<8.0.0,>=7.31.1->transformers-interpret) (2.0.10)
Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.8/dist-packages (from ipython<8.0.0,>=7.31.1->transformers-interpret) (57.4.0)
Requirement already satisfied: decorator in /usr/local/lib/python3.8/dist-packages (from ipython<8.0.0,>=7.31.1->transformers-interpret) (4.4.2)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.8/dist-packages (from ipython<8.0.0,>=7.31.1->transformers-interpret) (0.7.5)
Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers>=3.0.0->transformers-interpret) (3.9.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers>=3.0.0->transformers-interpret) (23.0)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers>=3.0.0->transformers-interpret) (4.64.1)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers>=3.0.0->transformers-interpret) (6.0)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
Downloading tokenizers-0.13.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 68.2 MB/s eta 0:00:00
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers>=3.0.0->transformers-interpret) (2022.6.2)
Collecting huggingface-hub<1.0,>=0.11.0
Downloading huggingface_hub-0.12.0-py3-none-any.whl (190 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 190.3/190.3 KB 14.2 MB/s eta 0:00:00
Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers>=3.0.0->transformers-interpret) (2.25.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers>=3.0.0->transformers-interpret) (4.4.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.8/dist-packages (from jedi>=0.16->ipython<8.0.0,>=7.31.1->transformers-interpret) (0.8.3)
Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.8/dist-packages (from pexpect>4.3->ipython<8.0.0,>=7.31.1->transformers-interpret) (0.7.0)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.8/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython<8.0.0,>=7.31.1->transformers-interpret) (1.15.0)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.8/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython<8.0.0,>=7.31.1->transformers-interpret) (0.2.6)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->captum>=0.3.1->transformers-interpret) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->captum>=0.3.1->transformers-interpret) (1.4.4)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->captum>=0.3.1->transformers-interpret) (2.8.2)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->captum>=0.3.1->transformers-interpret) (3.0.9)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers>=3.0.0->transformers-interpret) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers>=3.0.0->transformers-interpret) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers>=3.0.0->transformers-interpret) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers>=3.0.0->transformers-interpret) (2022.12.7)
Installing collected packages: tokenizers, matplotlib-inline, jedi, ipython, huggingface-hub, transformers, captum, transformers-interpret
Attempting uninstall: ipython
Found existing installation: ipython 7.9.0
Uninstalling ipython-7.9.0:
Successfully uninstalled ipython-7.9.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires ipython~=7.9.0, but you have ipython 7.34.0 which is incompatible.
Successfully installed captum-0.6.0 huggingface-hub-0.12.0 ipython-7.34.0 jedi-0.18.2 matplotlib-inline-0.1.6 tokenizers-0.13.2 transformers-4.26.0 transformers-interpret-0.9.6
Downloading (…)okenizer_config.json: 100%
26.0/26.0 [00:00<00:00, 1.07kB/s]
Downloading (…)lve/main/config.json: 100%
1.15k/1.15k [00:00<00:00, 60.9kB/s]
Downloading (…)olve/main/vocab.json: 100%
899k/899k [00:00<00:00, 2.73MB/s]
Downloading (…)olve/main/merges.txt: 100%
456k/456k [00:00<00:00, 1.48MB/s]
Downloading (…)/main/tokenizer.json: 100%
1.36M/1.36M [00:00<00:00, 3.66MB/s]
Downloading (…)"pytorch_model.bin";: 100%
1.63G/1.63G [00:06<00:00, 252MB/s]
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
[](https://localhost:8080/#) in
11
12
---> 13 word_attributions = zero_shot_explainer(
14 "Today apple released the new Macbook showing off a range of new features found in the proprietary silicon chip computer. ",
15 labels = ["finance", "technology", "sports"],

9 frames
[/usr/local/lib/python3.8/dist-packages/transformers_interpret/explainers/text/zero_shot_classification.py](https://localhost:8080/#) in __call__(self, text, labels, embedding_type, hypothesis_template, include_hypothesis, internal_batch_size, n_steps)
293 self.hypothesis_text = self.hypothesis_labels[i]
294 self.predicted_label = labels[i] + " (" + self.entailment_key.lower() + ")"
--> 295 super().__call__(
296 text,
297 class_name=self.entailment_key,

[/usr/local/lib/python3.8/dist-packages/transformers_interpret/explainers/text/sequence_classification.py](https://localhost:8080/#) in __call__(self, text, index, class_name, embedding_type, internal_batch_size, n_steps)
310 if internal_batch_size:
311 self.internal_batch_size = internal_batch_size
--> 312 return self._run(text, index, class_name, embedding_type=embedding_type)
313
314 def __str__(self):

[/usr/local/lib/python3.8/dist-packages/transformers_interpret/explainers/text/sequence_classification.py](https://localhost:8080/#) in _run(self, text, index, class_name, embedding_type)
264 self.text = self._clean_text(text)
265
--> 266 self._calculate_attributions(embeddings=embeddings, index=index, class_name=class_name)
267 return self.word_attributions # type: ignore
268

[/usr/local/lib/python3.8/dist-packages/transformers_interpret/explainers/text/zero_shot_classification.py](https://localhost:8080/#) in _calculate_attributions(self, embeddings, class_name, index)
201
202 reference_tokens = [token.replace("Ġ", "") for token in self.decode(self.input_ids)]
--> 203 lig = LIGAttributions(
204 self._forward,
205 embeddings,

[/usr/local/lib/python3.8/dist-packages/transformers_interpret/attributions.py](https://localhost:8080/#) in __init__(self, custom_forward, embeddings, tokens, input_ids, ref_input_ids, sep_id, attention_mask, target, token_type_ids, position_ids, ref_token_type_ids, ref_position_ids, internal_batch_size, n_steps)
49
50 if self.token_type_ids is not None and self.position_ids is not None:
---> 51 self._attributions, self.delta = self.lig.attribute(
52 inputs=(self.input_ids, self.token_type_ids, self.position_ids),
53 baselines=(

[/usr/local/lib/python3.8/dist-packages/captum/log/__init__.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
40 @wraps(func)
41 def wrapper(*args, **kwargs):
---> 42 return func(*args, **kwargs)
43
44 return wrapper

[/usr/local/lib/python3.8/dist-packages/captum/attr/_core/layer/layer_integrated_gradients.py](https://localhost:8080/#) in attribute(self, inputs, baselines, target, additional_forward_args, n_steps, method, internal_batch_size, return_convergence_delta, attribute_to_layer_input)
369 self.device_ids = getattr(self.forward_func, "device_ids", None)
370
--> 371 inputs_layer = _forward_layer_eval(
372 self.forward_func,
373 inps,

[/usr/local/lib/python3.8/dist-packages/captum/_utils/gradient.py](https://localhost:8080/#) in _forward_layer_eval(forward_fn, inputs, layer, additional_forward_args, device_ids, attribute_to_layer_input, grad_enabled)
180 grad_enabled: bool = False,
181 ) -> Union[Tuple[Tensor, ...], List[Tuple[Tensor, ...]]]:
--> 182 return _forward_layer_eval_with_neuron_grads(
183 forward_fn,
184 inputs,

[/usr/local/lib/python3.8/dist-packages/captum/_utils/gradient.py](https://localhost:8080/#) in _forward_layer_eval_with_neuron_grads(forward_fn, inputs, layer, additional_forward_args, gradient_neuron_selector, grad_enabled, device_ids, attribute_to_layer_input)
443
444 with torch.autograd.set_grad_enabled(grad_enabled):
--> 445 saved_layer = _forward_layer_distributed_eval(
446 forward_fn,
447 inputs,

[/usr/local/lib/python3.8/dist-packages/captum/_utils/gradient.py](https://localhost:8080/#) in _forward_layer_distributed_eval(forward_fn, inputs, layer, target_ind, additional_forward_args, attribute_to_layer_input, forward_hook_with_return, require_layer_grads)
303
304 if len(saved_layer) == 0:
--> 305 raise AssertionError("Forward hook did not obtain any outputs for given layer")
306
307 if forward_hook_with_return:

AssertionError: Forward hook did not obtain any outputs for given layer
```

This seems as minimal an example as possible, so I feel like something is afoot.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.