modelscope / modelscope/ms-swift
[RFC] Reward model (value-head RM) support in infer/deploy via vLLM
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Hi, I'd like to pick up this item from the #5721 roadmap: vLLM/SGLang support for classification models / RM.
Current state on main:
VllmEnginealready mapsembedding/seq_cls/rerankerto vLLM pooling tasks viaencode_task_mapping, so classification models are covered — but there is no task type for reward models.swift infer/swift deploycan't load a value-head RM directly, andsglang_engine.pyonly special-casesembedding.- The only ways to score an RM today are the in-process transformers forward in
DefaultRMPlugin(swift/rewards/rm_plugin.py), or deploying it separately with a barevllm serve.
Proposal:
- Add a
rewardtask type end-to-end: map it to the vLLM pooling runner inVllmEngineand return pooler scores frominfer. - Expose it in the infer/deploy CLIs so
swift deploy --task_type rewardworks out of the box. - Optionally, let GRPO's reward plugin call this engine, so the RM can run on separate GPUs with much higher throughput during training.
- SGLang support as a follow-up once the vLLM path is settled.
I'd like to work on this. Before I start — does this scope look right, and would reward be the preferred task_type name? Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading VllmEngine's encode_task_mapping and the existing pooling-task handling, then inspect swift infer/deploy task-type definitions and sglang_engine.py. Compare these paths with swift/rewards/rm_plugin.py to define the smallest vLLM-only scope. Done should mean the agreed reward task works through infer and deploy, with CLI behavior and tests covering the supported path; SGLang and GRPO integration are explicitly resolved as in- or out-of-scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100