cdpierse / cdpierse/transformers-interpret

Binary Classification: How is predicted label computed?

Open
#98 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.4k
Forks
99
PR merge metrics
No merged PRs in 30d

Description

Hi there,

I am observing the following (strange) behavior when using `pipeline` from the `transformers` library and `transformer-interpret`:

```
text = "Now Accord networks is a company in video, and he led the sales team, and the marketing group at Accord, and he took it from start up, sound familiar, it's from start up to $60 million company in two years."
classifier = pipeline('text-classification', model=model, tokenizer=tokenizer, device=0)
classifier(text)
```
```
[{'label': 'LABEL_1', 'score': 0.9711543321609497}]
```

while `transformer-interpret` gives me slightly different scores:
```
explainer = SequenceClassificationExplainer(model, tokenizer)
attributions = explainer(text)
html = explainer.visualize()
```
![image](https://user-images.githubusercontent.com/53626067/183944745-cbe494c6-aa00-4048-a609-18556d4e887d.png)

In both cases I apply the exact same `model` and `tokenizer`...

I am grateful for any hint and/or advice! 🤗

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the shown `pipeline` and `SequenceClassificationExplainer` calls with the same `model`, `tokenizer`, and text. Compare how each computes and reports the predicted label and score, then document or correct the discrepancy and verify the result against both outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.