lm-sys / lm-sys/FastChat

Include model context window limit on the API endpoint?

Open
#1,871 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

I am developing a document summary application in which the user can dynamically select various model endpoints from `fastchat.serve.controller`.

Since document chunking depends entirely on the available context window of the model, it would be good if this parameter (e.g. something like `max_position_embeddings` for the vicuna-7b-v1.3 model) could be exposed on the API `/models` endpoint.

On a related note:

`curl http://localhost:8000/v1/models`

returns useful information, e.g.

`{"object":"list","data":[{"id":"vicuna-7b-v1.3","object":"model","created":1688626343,"owned_by":"fastchat","root":"vicuna-7b-v1.3","parent":null,"permission":[{"id":"modelperm-3Ki2CyP8sjBPGfZbtDDYqK","object":"model_permission","created":1688626343,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":true,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}`

but, e.g.,

`curl http://localhost:8000/v1/models/vicuna-7b-v1.3`

returns

`{"detail":"Not Found"}`

Is this a bug? According to the OpenAI API specification, this endpoint should return specific model parameters.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fastchat.serve.controller and inspect the existing /v1/models route and its response shape. Compare the requested model-specific endpoint and context-window metadata with the OpenAI API specification; done means the model information is exposed consistently and the shown curl requests return the expected details.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.