huggingface / huggingface/lighteval

[BUG] Unable to Download Dataset in Restricted Network Environment (No Access to Hugging Face or Mirrors)

Open
#802 10 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
In restricted or offline network environments, it is currently not possible to run `lighteval` because it attempts to download datasets (e.g., from the Hugging Face Hub or mirrors) without fallback options. When access to sites like `https://huggingface.co` or `https://hf-mirror.com` is blocked, dataset loading fails completely.

## To Reproduce
1. Run `lighteval` in an environment without internet access or with Hugging Face domains blocked.
2. Observe that dataset loading (e.g., via `load_dataset()` from the `datasets` library) fails.
3. Example error:

```
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='huggingface.co', ...)
```

## Expected behavior
* Support for using pre-downloaded datasets from local disk (e.g., via environment variables or explicit local paths).
* Graceful handling of offline environments or clear documentation for air-gapped usage.

## Additional Context

* Many research or production environments have strict firewall policies or no internet access.
* This significantly limits the adoption of `lighteval` in such contexts.
* A partial workaround is to manually preload datasets into the Hugging Face `datasets` cache, but this is not natively supported by `lighteval`.

I have implemented support for running `lighteval` in fully offline mode and plan to submit a PR soon.
Hopefully, this will help address the issue and make the tool more accessible in restricted environments.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.