huggingface / huggingface/lighteval

[BUG] Issue with CACHE_DIR Default Value in Accelerate Pipeline

Open
#460 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
The default value for CACHE_DIR in the pipeline creates several issues. On many systems, such as Ubuntu distributions, users typically lack permissions for the root directory (`/`), which prevents the pipeline from running. On Windows, it creates a `scratch` folder on the C drive, which is not ideal.

https://github.com/huggingface/lighteval/blob/0977fde60ba3564cf34c53606a53962d81eef7b0/src/lighteval/main_accelerate.py#L34

## Proposed Solution
Remove the default value for CACHE_DIR and instead return None. This will allow the pipeline to follow the default logic of the transformers library.

``` python
CACHE_DIR: str = os.getenv("HF_HOME")
```

## To Reproduce
Run the example provided in the README on any Linux distribution:

``` sh
HF_HOME= lighteval accelerate \
"pretrained=gpt2" \
"leaderboard|truthfulqa:mc|0|0"
```

## Expected behavior
The running should execute without errors.

## Version info
The issue was encountered on the `main` branch installed with `pip install -e .[dev]`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.