[Feature] Can we support parameter n in OpenAI compatible API?
Open
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Motivation
In /v1/completions and /v1/chat/completions endpoint, can we support the parameter `n`?
So that we can sampling multiple outputs for the same input.
Currently, we can only call the endpoint multiple times which is not efficient.
### Related resources
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start at the OpenAI-compatible /v1/completions and /v1/chat/completions endpoint implementations and trace how request parameters and generated outputs are handled. Done means both endpoints accept n and return multiple sampled outputs for one input without requiring repeated client calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100