open-compass / open-compass/opencompass
[Feature] Add another OpenAISDK model using prompt completion API
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
Describe the feature
For now, OpenAISDK model only supports chat/completion API, which makes the infer service applies a chat template and triggers CoT, in its _generate method. CoT takes a large number of tokens, slows down the inference speed (request per minutes), and increases the probability of a answer got truncated.
To fix this, I request to add a new model, based on the original OpenAISDK, but using completion API instead of chat/completion. The completion API will only do completion by the prompt, and will give out a much high better result in simple eval tasks like choices, true/false, etc. The completion API doesn't take in messages and roles, so it cannot be used for complicated tasks like function call or multi-turn.
Will you implement it?
- I would like to implement this feature and create a PR!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing OpenAISDK model and its _generate method in the repository, then trace how the infer service applies chat templates and passes messages to chat/completion. Compare that flow with the completion API requirements and existing model tests or evaluation entry points. Done means a separately usable model supports prompt completion while preserving the existing model's chat and multi-turn behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100