InternLM / InternLM/lmdeploy

[Feature] 使用lmdeploy里的pipeline推理可以关掉qwen3的推理模式吗

Open
#4,010 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Motivation

推理代码如下:

```
class Inference:
def __init__(self, model_path, template_name='qwen3'):
backend_config = TurbomindEngineConfig(tp=1,device_name='cpu',max_batch_size=1, model_format='awq')
chat_template_config = ChatTemplateConfig(model_name="qwen3")
self.pipe = pipeline(model_path,chat_template_config=chat_template_config,
backend_config=backend_config,device='cpu')
# self.pipe = pipeline(model_path)

def infer(self, query, threshold=5):
gen_config = GenerationConfig(max_new_tokens=1024,
output_logits="generation")

response = self.pipe([query],
gen_config=gen_config
)
print("response:", response)
outputs_new = response[0]

logits = outputs_new.logits
return logits
```

主要有两个问题:
1.为什么加上 backend_config和chat_template_config后pipeline加载本地微调后模型会报错“ValueError: Try apply_chat_template failed: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: ''.”,但是不加这两个config的话是能够有推理结果的,但是和直接使用AutoModelForCausalLM推理返回的logits结果差异很大,该怎么修改呢

2.应该在哪个config里面显式关掉thinking模式

### Related resources

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.