Include model context window limit on the API endpoint?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
I am developing a document summary application in which the user can dynamically select various model endpoints from `fastchat.serve.controller`.
Since document chunking depends entirely on the available context window of the model, it would be good if this parameter (e.g. something like `max_position_embeddings` for the vicuna-7b-v1.3 model) could be exposed on the API `/models` endpoint.
On a related note:
`curl http://localhost:8000/v1/models`
returns useful information, e.g.
`{"object":"list","data":[{"id":"vicuna-7b-v1.3","object":"model","created":1688626343,"owned_by":"fastchat","root":"vicuna-7b-v1.3","parent":null,"permission":[{"id":"modelperm-3Ki2CyP8sjBPGfZbtDDYqK","object":"model_permission","created":1688626343,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":true,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}`
but, e.g.,
`curl http://localhost:8000/v1/models/vicuna-7b-v1.3`
returns
`{"detail":"Not Found"}`
Is this a bug? According to the OpenAI API specification, this endpoint should return specific model parameters.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fastchat.serve.controller and inspect the existing /v1/models route and its response shape. Compare the requested model-specific endpoint and context-window metadata with the OpenAI API specification; done means the model information is exposed consistently and the shown curl requests return the expected details.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100