huggingface / huggingface/lighteval
[BUG] Handle BERTScore baseline files
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
The default BERTScorer has a `rescale_with_baseline` argument that is set to `True`. When this flag is set, the `BERTScorer` class requires a `baseline_path` to be set, otherwise[ it will attempt to load TSV files](https://github.com/huggingface/lighteval/blob/main/src/lighteval/metrics/imports/bert_scorer.py#L376) that are expected to be in the `src/lighteval/metrics/imports` directory.
## To Reproduce
Requires #990 to be applied to fix overflow error.
```python
from lighteval.models.dummy.dummy_model import DummyModelConfig
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters
def main():
task = "helm|summarization:xsum|0"
model_config = DummyModelConfig(
model_name="dummy",
seed=1234,
)
pipeline_params = PipelineParameters(
launcher_type=ParallelismManager.NONE,
max_samples=1
)
evaluation_tracker = EvaluationTracker(
output_dir="./results",
save_details=True,
push_to_hub=False,
)
pipeline = Pipeline(
tasks=task,
pipeline_parameters=pipeline_params,
evaluation_tracker=evaluation_tracker,
model_config=model_config,
)
pipeline.evaluate()
if __name__ == "__main__":
main()
```
## Expected behavior
Either a warning is displayed to the user to let them know that the baselines need to be manually downloaded and where they can be acquired, or lighteval will handle the management of loading the baseline files. It looks like the files are downloaded from [here](https://github.com/Tiiiger/bert_score/tree/master/bert_score/rescale_baseline).
I'm happy to create a PR to fix this, but need to know the preferred method.
## Version info
`main` on `c7a063aee58dc14f51841a6b4f977cd914e10a8f`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.