huggingface / huggingface/lighteval
[BUG] Issue with CACHE_DIR Default Value in Accelerate Pipeline
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
The default value for CACHE_DIR in the pipeline creates several issues. On many systems, such as Ubuntu distributions, users typically lack permissions for the root directory (`/`), which prevents the pipeline from running. On Windows, it creates a `scratch` folder on the C drive, which is not ideal.
https://github.com/huggingface/lighteval/blob/0977fde60ba3564cf34c53606a53962d81eef7b0/src/lighteval/main_accelerate.py#L34
## Proposed Solution
Remove the default value for CACHE_DIR and instead return None. This will allow the pipeline to follow the default logic of the transformers library.
``` python
CACHE_DIR: str = os.getenv("HF_HOME")
```
## To Reproduce
Run the example provided in the README on any Linux distribution:
``` sh
HF_HOME= lighteval accelerate \
"pretrained=gpt2" \
"leaderboard|truthfulqa:mc|0|0"
```
## Expected behavior
The running should execute without errors.
## Version info
The issue was encountered on the `main` branch installed with `pip install -e .[dev]`.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.