modelscope / modelscope/ms-swift

[RFC] Reward model (value-head RM) support in infer/deploy via vLLM

Open
#10,058 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Hi, I'd like to pick up this item from the #5721 roadmap: vLLM/SGLang support for classification models / RM.

Current state on main:

  • VllmEngine already maps embedding / seq_cls / reranker to vLLM pooling tasks via encode_task_mapping, so classification models are covered — but there is no task type for reward models.
  • swift infer / swift deploy can't load a value-head RM directly, and sglang_engine.py only special-cases embedding.
  • The only ways to score an RM today are the in-process transformers forward in DefaultRMPlugin (swift/rewards/rm_plugin.py), or deploying it separately with a bare vllm serve.

Proposal:

  1. Add a reward task type end-to-end: map it to the vLLM pooling runner in VllmEngine and return pooler scores from infer.
  2. Expose it in the infer/deploy CLIs so swift deploy --task_type reward works out of the box.
  3. Optionally, let GRPO's reward plugin call this engine, so the RM can run on separate GPUs with much higher throughput during training.
  4. SGLang support as a follow-up once the vLLM path is settled.

I'd like to work on this. Before I start — does this scope look right, and would reward be the preferred task_type name? Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading VllmEngine's encode_task_mapping and the existing pooling-task handling, then inspect swift infer/deploy task-type definitions and sglang_engine.py. Compare these paths with swift/rewards/rm_plugin.py to define the smallest vLLM-only scope. Done should mean the agreed reward task works through infer and deploy, with CLI behavior and tests covering the supported path; SGLang and GRPO integration are explicitly resolved as in- or out-of-scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.