huggingface / huggingface/lighteval
[BUG] Unable to Download Dataset in Restricted Network Environment (No Access to Hugging Face or Mirrors)
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
In restricted or offline network environments, it is currently not possible to run `lighteval` because it attempts to download datasets (e.g., from the Hugging Face Hub or mirrors) without fallback options. When access to sites like `https://huggingface.co` or `https://hf-mirror.com` is blocked, dataset loading fails completely.
## To Reproduce
1. Run `lighteval` in an environment without internet access or with Hugging Face domains blocked.
2. Observe that dataset loading (e.g., via `load_dataset()` from the `datasets` library) fails.
3. Example error:
```
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='huggingface.co', ...)
```
## Expected behavior
* Support for using pre-downloaded datasets from local disk (e.g., via environment variables or explicit local paths).
* Graceful handling of offline environments or clear documentation for air-gapped usage.
## Additional Context
* Many research or production environments have strict firewall policies or no internet access.
* This significantly limits the adoption of `lighteval` in such contexts.
* A partial workaround is to manually preload datasets into the Hugging Face `datasets` cache, but this is not natively supported by `lighteval`.
I have implemented support for running `lighteval` in fully offline mode and plan to submit a PR soon.
Hopefully, this will help address the issue and make the tool more accessible in restricted environments.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.