huggingface / huggingface/setfit

Methodological error in zero cost, zero time, zero shot notebook

Open
#511 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.8k
Forks
267
Avg merge
36m
Merged PRs (30d)
5

Description

Hi,

I was looking at the zero cost, zero time, zero shot notebook for financial sentiment analysis (i.e., [this one](https://github.com/huggingface/setfit/blob/main/notebooks/zero_cost_zero_time_zero_shot_financial_sentiment_analysis.ipynb)), and discovered a methodological error that invalidates the conclusions of the distillation section.

What happens is that the train and test dataframes, i.e., the CSV files loaded from Moritz Laurer's blog, are created by splitting the _train_ split of the dataset (the dataset doesn't have a test split). Later on, when distilling, the authors of blog post reload the _entire_ train split of the dataset, and then use this to distill the MLP. This means that the test data is also used to distill the model, which leads to a big overestimation of performance.

In my experiments, the original score PRF score I got was:

```
(array([0.85507246, 0.97348485, 0.94166667]),
array([0.96721311, 0.96981132, 0.88976378]),
array([0.90769231, 0.97164461, 0.91497976]),
array([ 61, 265, 127]))
```

Which is close to the reported score in the article.
If I instead remove the test data from the data used to distill the MLP, I get much lower scores:

```
(array([0.76785714, 0.87632509, 0.78947368]),
array([0.70491803, 0.93584906, 0.70866142]),
array([0.73504274, 0.90510949, 0.74688797]),
array([ 61, 265, 127]))
```

These scores are much lower than the reported scores, and also much lower than the LLM scores, which invalidates the conclusion of the notebook and article. Note that these scores are still a bit higher than the scores you would get when just directly optimizing cross entropy, so you could argue that the point still makes sense.

If you want I can do a PR on the notebook.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.