huggingface / huggingface/lighteval

[BUG] Handle BERTScore baseline files

Open
#995 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
The default BERTScorer has a `rescale_with_baseline` argument that is set to `True`. When this flag is set, the `BERTScorer` class requires a `baseline_path` to be set, otherwise[ it will attempt to load TSV files](https://github.com/huggingface/lighteval/blob/main/src/lighteval/metrics/imports/bert_scorer.py#L376) that are expected to be in the `src/lighteval/metrics/imports` directory.

## To Reproduce
Requires #990 to be applied to fix overflow error.

```python
from lighteval.models.dummy.dummy_model import DummyModelConfig
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters

def main():
task = "helm|summarization:xsum|0"

model_config = DummyModelConfig(
model_name="dummy",
seed=1234,
)

pipeline_params = PipelineParameters(
launcher_type=ParallelismManager.NONE,
max_samples=1
)

evaluation_tracker = EvaluationTracker(
output_dir="./results",
save_details=True,
push_to_hub=False,
)

pipeline = Pipeline(
tasks=task,
pipeline_parameters=pipeline_params,
evaluation_tracker=evaluation_tracker,
model_config=model_config,
)
pipeline.evaluate()

if __name__ == "__main__":
main()
```

## Expected behavior
Either a warning is displayed to the user to let them know that the baselines need to be manually downloaded and where they can be acquired, or lighteval will handle the management of loading the baseline files. It looks like the files are downloaded from [here](https://github.com/Tiiiger/bert_score/tree/master/bert_score/rescale_baseline).

I'm happy to create a PR to fix this, but need to know the preferred method.

## Version info
`main` on `c7a063aee58dc14f51841a6b4f977cd914e10a8f`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.