lm-sys / lm-sys/FastChat

Document diverged OpenAI API

Open
#3,147 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

The existence of the diverged OpenAI API appears to be undocumented at the moment, but has some useful and requested features (e.g. `repetition_penalty` #1914).

As far as I can tell, it appears these endpoints can be used with the OpenAI client, and additional parameters passed with the `extra_body` argument, e.g.:

```python
# ...
openai.base_url = "http://localhost:8000/api/v1/"
# ...
completion = openai.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Hello! What is your name?"}],
extra_body={
"repetition_penalty": 1.10
}
)
```

(this is also useful for `top_k` which is not a supported argument for the OpenAI client but is supported in both APIs in FastChat)

It would be good to at least document that this API exists and confirm whether the divergences actually impact using the OpenAI client.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating FastChat's OpenAI-compatible API documentation and serving entry points, then compare the documented endpoints with the OpenAI client example in the issue. Document that the diverged API exists and clarify whether extra_body supports parameters such as repetition_penalty and top_k.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.