huggingface / huggingface/lighteval

[BUG] lighteval does not support openai endpoints but says it does

Open
#960 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
In the README and when running help, there are references to openai endpoint support, but it appears to no longer exist.
```
$ lighteval endpoint --help

Usage: lighteval endpoint [OPTIONS] COMMAND [ARGS]...

Evaluate models using some endpoint (tgi, inference endpoint, openai) as backend.

╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Evaluation Backends ────────────────────────────────────────────────────────────────────────────────╮
│ inference-endpoint Evaluate models using inference-endpoints as backend. │
│ tgi Evaluate models using TGI as backend. │
│ litellm Evaluate models using LiteLLM as backend. │
│ inference-providers Evaluate models using LiteLLM as backend. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

```
$ lighteval endpoint openai --help
Usage: lighteval endpoint [OPTIONS] COMMAND [ARGS]...
Try 'lighteval endpoint --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such command 'openai'. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
```

## Expected behavior
If the openai/openai-compatible support was removed, please remove it from your documentation or indicate clearly that it is no longer supported.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.