huggingface / huggingface/lighteval
OpenAI-compat via endpoint litellm: openai/ prefix and /v1 base_url
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Working config (endpoint litellm, openai/ prefix, /v1)
lighteval already evaluates OpenAI-compatible hosts through `lighteval endpoint litellm`. A Chat Completions host whose public catalog is `GET https://api.pzero.studio/v1/models` (no key) fits that backend.
```bash
export OPENAI_API_KEY=
lighteval endpoint litellm \
"provider=openai,model_name=openai/deepseek-v4-flash,base_url=https://api.pzero.studio/v1" \
gsm8k --max-samples 1
```
Same join as YAML `model_parameters`: `provider: openai`, `model_name: openai/`, `base_url: https://api.pzero.studio/v1`.
`base_url` is the `/v1` root. LiteLLM appends `/chat/completions`. Do not paste the chat path. `model_name` needs the `openai/` prefix.
Do not use `lighteval endpoint tgi`, `inference-endpoint`, or `vllm` for this host. Do not use `provider: hosted_vllm` unless the process is actually vLLM.
Not asking for a first-party provider id. `endpoint litellm` plus `openai/` is enough.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the `lighteval endpoint litellm` entry point and the `model_parameters` YAML configuration mentioned in the issue. Verify the provided gsm8k command with the `/v1` base URL and `openai/` model prefix; done means the supported configuration and exclusions are documented clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- yaml
- Domain
- cli, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100