[Feature] 使用lmdeploy里的pipeline推理可以关掉qwen3的推理模式吗
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Motivation
推理代码如下:
```
class Inference:
def __init__(self, model_path, template_name='qwen3'):
backend_config = TurbomindEngineConfig(tp=1,device_name='cpu',max_batch_size=1, model_format='awq')
chat_template_config = ChatTemplateConfig(model_name="qwen3")
self.pipe = pipeline(model_path,chat_template_config=chat_template_config,
backend_config=backend_config,device='cpu')
# self.pipe = pipeline(model_path)
def infer(self, query, threshold=5):
gen_config = GenerationConfig(max_new_tokens=1024,
output_logits="generation")
response = self.pipe([query],
gen_config=gen_config
)
print("response:", response)
outputs_new = response[0]
logits = outputs_new.logits
return logits
```
主要有两个问题:
1.为什么加上 backend_config和chat_template_config后pipeline加载本地微调后模型会报错“ValueError: Try apply_chat_template failed: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: ''.”,但是不加这两个config的话是能够有推理结果的,但是和直接使用AutoModelForCausalLM推理返回的logits结果差异很大,该怎么修改呢
2.应该在哪个config里面显式关掉thinking模式
### Related resources
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.