huggingface / huggingface/lighteval

OpenAI-compat via endpoint litellm: openai/ prefix and /v1 base_url

Open
#1,362 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Working config (endpoint litellm, openai/ prefix, /v1)

lighteval already evaluates OpenAI-compatible hosts through `lighteval endpoint litellm`. A Chat Completions host whose public catalog is `GET https://api.pzero.studio/v1/models` (no key) fits that backend.

```bash
export OPENAI_API_KEY=
lighteval endpoint litellm \
"provider=openai,model_name=openai/deepseek-v4-flash,base_url=https://api.pzero.studio/v1" \
gsm8k --max-samples 1
```

Same join as YAML `model_parameters`: `provider: openai`, `model_name: openai/`, `base_url: https://api.pzero.studio/v1`.

`base_url` is the `/v1` root. LiteLLM appends `/chat/completions`. Do not paste the chat path. `model_name` needs the `openai/` prefix.

Do not use `lighteval endpoint tgi`, `inference-endpoint`, or `vllm` for this host. Do not use `provider: hosted_vllm` unless the process is actually vLLM.

Not asking for a first-party provider id. `endpoint litellm` plus `openai/` is enough.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the `lighteval endpoint litellm` entry point and the `model_parameters` YAML configuration mentioned in the issue. Verify the provided gsm8k command with the `/v1` base URL and `openai/` model prefix; done means the supported configuration and exclusions are documented clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
yaml
Domain
cli, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.