huggingface / huggingface/lighteval
Append revision to filepath in `--output_dir`?
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Currently, `lighteval` stores results/details in a path that is determined by the model name, e.g.
```shell
scratch/evals
├── details
│ └── Qwen
│ └── Qwen1.5-0.5B-Chat
│ ├── 2024-02-26T15-36-31.681219
│ │ └── details_lighteval|truthfulqa:mc|0_2024-02-26T15-36-31.681219.parquet
│ └── results_2024-02-26T15-36-31.681219.json
└── results
└── Qwen
└── Qwen1.5-0.5B-Chat
└── results_2024-02-26T15-36-31.681219.json
```
However, I am quite often evaluating models with different _revisions_ and the current save logic groups these all together in the same subfolder which makes it hard to determine which result corresponds to which run.
Would it make sense to append the model revision parameter to the filepaths, e.g. something like this for the `main` revision (or whatever is passed to the `revision` arg in the script):
```shell
scratch/evals
├── details
│ └── Qwen
│ └── Qwen1.5-0.5B-Chat
│ └── main
│ ├── 2024-02-26T15-36-31.681219
│ │ └── details_lighteval|truthfulqa:mc|0_2024-02-26T15-36-31.681219.parquet
│ └── results_2024-02-26T15-36-31.681219.json
└── results
└── Qwen
└── Qwen1.5-0.5B-Chat
└── main
└── results_2024-02-26T15-36-31.681219.json
```
My current workaround is to manually specify the model path in `--output_dir={ORG}/{MODEL_ID}/{REVISION}` and then glob the files. This is fine, but a bit clunky because one ends up with a long nested path like `{ORG}/{MODEL_ID}/{REVISION}/results/{ORG}/{MODEL_ID}`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.