huggingface / huggingface/evaluate

DownloadConfig is not passed to LocalEvaluationModuleFactory when loading from local file, causing uncontrollable download behavior

Open
#709 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

In evaluate/src/evaluate/loading.py lines 621-624:

```python
elif os.path.isfile(combined_path):
return LocalEvaluationModuleFactory(
combined_path, download_mode=download_mode, dynamic_modules_path=dynamic_modules_path
).get_module()
```

The `DownloadConfig` from `evaluation_module_factory` is not passed to `LocalEvaluationModuleFactory`. This causes line 421:

```python
local_imports = _download_additional_modules(
name=self.name,
base_path=str(Path(self.path).parent),
imports=imports,
download_config=self.download_config,
)
```

to use the default `download_config`. Consequently, in `_download_additional_modules` (line 248), if `url_or_filename` is an external URL, it will use the default `download_config` when entering the `cached_path` function.

Subsequently, when entering the `get_from_cache` function in `evaluate/src/evaluate/utils/file_utils.py`, `use_etag=True` and `local_files_only=False`. This skips line 438:

```python
if os.path.exists(cache_path) and not force_download and not use_etag:
```

because `use_etag=True`. Then it proceeds to the `if not local_files_only` branch on line 447. If my server can't connect to Hugging Face at this point, it will get stuck here.

I also can't modify the download configuration by changing the `DownloadConfig` in the `load` function because it's not passed to the `LocalEvaluationModuleFactory` function. Nor can I use already downloaded files in the cache because the line `if os.path.exists(cache_path) and not force_download and not use_etag:` is skipped.

Is this a bug? If not, how can I use the `evaluate.load` function when connections to the Hugging Face server are prone to timeouts?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.