huggingface / huggingface/evaluate
DownloadConfig is not passed to LocalEvaluationModuleFactory when loading from local file, causing uncontrollable download behavior
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
In evaluate/src/evaluate/loading.py lines 621-624:
```python
elif os.path.isfile(combined_path):
return LocalEvaluationModuleFactory(
combined_path, download_mode=download_mode, dynamic_modules_path=dynamic_modules_path
).get_module()
```
The `DownloadConfig` from `evaluation_module_factory` is not passed to `LocalEvaluationModuleFactory`. This causes line 421:
```python
local_imports = _download_additional_modules(
name=self.name,
base_path=str(Path(self.path).parent),
imports=imports,
download_config=self.download_config,
)
```
to use the default `download_config`. Consequently, in `_download_additional_modules` (line 248), if `url_or_filename` is an external URL, it will use the default `download_config` when entering the `cached_path` function.
Subsequently, when entering the `get_from_cache` function in `evaluate/src/evaluate/utils/file_utils.py`, `use_etag=True` and `local_files_only=False`. This skips line 438:
```python
if os.path.exists(cache_path) and not force_download and not use_etag:
```
because `use_etag=True`. Then it proceeds to the `if not local_files_only` branch on line 447. If my server can't connect to Hugging Face at this point, it will get stuck here.
I also can't modify the download configuration by changing the `DownloadConfig` in the `load` function because it's not passed to the `LocalEvaluationModuleFactory` function. Nor can I use already downloaded files in the cache because the line `if os.path.exists(cache_path) and not force_download and not use_etag:` is skipped.
Is this a bug? If not, how can I use the `evaluate.load` function when connections to the Hugging Face server are prone to timeouts?
Contributor guide
Assessment
This issue has not been assessed yet.