open-compass / open-compass/opencompass

[Feature] Add another OpenAISDK model using prompt completion API

Open
#2,386 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

Describe the feature

For now, OpenAISDK model only supports chat/completion API, which makes the infer service applies a chat template and triggers CoT, in its _generate method. CoT takes a large number of tokens, slows down the inference speed (request per minutes), and increases the probability of a answer got truncated.

To fix this, I request to add a new model, based on the original OpenAISDK, but using completion API instead of chat/completion. The completion API will only do completion by the prompt, and will give out a much high better result in simple eval tasks like choices, true/false, etc. The completion API doesn't take in messages and roles, so it cannot be used for complicated tasks like function call or multi-turn.

Will you implement it?
  • I would like to implement this feature and create a PR!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing OpenAISDK model and its _generate method in the repository, then trace how the infer service applies chat templates and passes messages to chat/completion. Compare that flow with the completion API requirements and existing model tests or evaluation entry points. Done means a separately usable model supports prompt completion while preserving the existing model's chat and multi-turn behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.