InternLM / InternLM/lmdeploy

[Feature] Can we support parameter n in OpenAI compatible API?

Open
#4,073 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Motivation

In /v1/completions and /v1/chat/completions endpoint, can we support the parameter `n`?

So that we can sampling multiple outputs for the same input.

Currently, we can only call the endpoint multiple times which is not efficient.

### Related resources

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at the OpenAI-compatible /v1/completions and /v1/chat/completions endpoint implementations and trace how request parameters and generated outputs are handled. Done means both endpoints accept n and return multiple sampled outputs for one input without requiring repeated client calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.