huggingface / huggingface/lighteval

Append revision to filepath in `--output_dir`?

Open
#56 6 comments 0 reactions 0 assignees View on GitHub
feature good first issue science-team
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Currently, `lighteval` stores results/details in a path that is determined by the model name, e.g.

```shell
scratch/evals
├── details
│ └── Qwen
│ └── Qwen1.5-0.5B-Chat
│ ├── 2024-02-26T15-36-31.681219
│ │ └── details_lighteval|truthfulqa:mc|0_2024-02-26T15-36-31.681219.parquet
│ └── results_2024-02-26T15-36-31.681219.json
└── results
└── Qwen
└── Qwen1.5-0.5B-Chat
└── results_2024-02-26T15-36-31.681219.json
```

However, I am quite often evaluating models with different _revisions_ and the current save logic groups these all together in the same subfolder which makes it hard to determine which result corresponds to which run.

Would it make sense to append the model revision parameter to the filepaths, e.g. something like this for the `main` revision (or whatever is passed to the `revision` arg in the script):

```shell
scratch/evals
├── details
│ └── Qwen
│ └── Qwen1.5-0.5B-Chat
│ └── main
│ ├── 2024-02-26T15-36-31.681219
│ │ └── details_lighteval|truthfulqa:mc|0_2024-02-26T15-36-31.681219.parquet
│ └── results_2024-02-26T15-36-31.681219.json
└── results
└── Qwen
└── Qwen1.5-0.5B-Chat
└── main
└── results_2024-02-26T15-36-31.681219.json
```

My current workaround is to manually specify the model path in `--output_dir={ORG}/{MODEL_ID}/{REVISION}` and then glob the files. This is fine, but a bit clunky because one ends up with a long nested path like `{ORG}/{MODEL_ID}/{REVISION}/results/{ORG}/{MODEL_ID}`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.