huggingface / huggingface/evaluate
Unable to use BLEURT in offline mode
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
Trying to use [BLEURT](https://github.com/huggingface/datasets/tree/main/metrics/bleurt) in offline mode fails. The script and model weights are cached to disk fine (when in _online_ mode). In _offline_ mode, it loads the script from the cache fine, but when trying to load the cached model weights, it throws an error.
I looks like the bug exists somewhere in the `get_from_cache` function, as the error is thrown from here:
https://github.com/huggingface/datasets/blob/f96547708a889c09ca8a02ed7aadd8c5690503c5/src/datasets/utils/file_utils.py#L530
> I know the metrics within `datasets` are deprecated. However, this exact error is thrown by [`evaluate`](https://github.com/huggingface/evaluate) as well.
### Steps to reproduce the bug
Steps to reproduce the behaviour:
```python
from datasets import load_metric
import os
os.environ["HF_DATASETS_OFFLINE"] = "1"
bleurt = load_metric("bleurt")
```
Gives the following error:
```shell
Traceback (most recent call last):
File "", line 1, in
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/utils/deprecation_utils.py", line 46, in wrapper
return deprecated_function(*args, **kwargs)
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/load.py", line 1397, in load_metric
metric.download_and_prepare(download_config=download_config)
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/metric.py", line 625, in download_and_prepare
self._download_and_prepare(dl_manager)
File "/home/johnmg/.cache/huggingface/modules/datasets_modules/metrics/bleurt/89f7c298fa543e9cee6749e6ed198069d7c10fc8e99c0ff37a843dbc0eea88d7/bleurt.py", line 117, in _download_and_prepare
model_path = dl_manager.download_and_extract(CHECKPOINT_URLS[checkpoint_name])
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/download/download_manager.py", line 564, in download_and_extract
return self.extract(self.download(url_or_urls))
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/download/download_manager.py", line 427, in download
downloaded_path_or_paths = map_nested(
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 436, in map_nested
return function(data_struct)
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/download/download_manager.py", line 453, in _download
return cached_path(url_or_filename, download_config=download_config)
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 182, in cached_path
output_path = get_from_cache(
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 530, in get_from_cache
_raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
File "/home/johnmg/mediqa/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 260, in _raise_if_offline_mode_is_enabled
raise OfflineModeIsEnabled(
datasets.utils.file_utils.OfflineModeIsEnabled: Offline mode is enabled. Tried to reach https://storage.googleapis.com/bleurt-oss/bleurt-base-128.zip
```
### Expected behavior
I would expect that, after loading the metric as `bleurt = load_metric("bleurt")` _with_ an internet connection it will be cached locally, and I should be able to load it from this cache _without_ an internet connection afterwards. I also considered manually specifying the cached model filepath like so:
```python
bleurt = load_metric("bleurt", "/home/johnmg/.cache/huggingface/metrics/bleurt/default/downloads/extracted/4686726448df12b97ad0880ca1f80735f419854eb56f1878cf550dcbd717fb20/bleurt-base-128")
```
but this doesn't work either:
```shell
KeyError: "/home/johnmg/.cache/huggingface/metrics/bleurt/default/downloads/extracted/4686726448df12b97ad0880ca1f80735f419854eb56f1878cf550dcbd717fb20/bleurt-base-128 model not found. You should supply the name of a model checkpoint for bleurt in dict_keys(['bleurt-tiny-128', 'bleurt-tiny-512', 'bleurt-base-128', 'bleurt-base-512', 'bleurt-large-128', 'bleurt-large-512', 'BLEURT-20-D3', 'BLEURT-20-D6', 'BLEURT-20-D12', 'BLEURT-20'])"
```
as the metric loading scripts expect the model checkpoint to be one of:
https://github.com/huggingface/datasets/blob/f96547708a889c09ca8a02ed7aadd8c5690503c5/metrics/bleurt/bleurt.py#L64-L75
### Environment info
I installed datasets from main with `pip install git+https://github.com/huggingface/datasets.git`
Contributor guide
Assessment
This issue has not been assessed yet.