huggingface / huggingface/lighteval
[BUG] lighteval does not support openai endpoints but says it does
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
In the README and when running help, there are references to openai endpoint support, but it appears to no longer exist.
```
$ lighteval endpoint --help
Usage: lighteval endpoint [OPTIONS] COMMAND [ARGS]...
Evaluate models using some endpoint (tgi, inference endpoint, openai) as backend.
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Evaluation Backends ────────────────────────────────────────────────────────────────────────────────╮
│ inference-endpoint Evaluate models using inference-endpoints as backend. │
│ tgi Evaluate models using TGI as backend. │
│ litellm Evaluate models using LiteLLM as backend. │
│ inference-providers Evaluate models using LiteLLM as backend. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
```
$ lighteval endpoint openai --help
Usage: lighteval endpoint [OPTIONS] COMMAND [ARGS]...
Try 'lighteval endpoint --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such command 'openai'. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
## Expected behavior
If the openai/openai-compatible support was removed, please remove it from your documentation or indicate clearly that it is no longer supported.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.