modelscope / modelscope/ms-swift

GME模型分别用vLLM部署和swift部署(使用vllm backend )的服务,返回的Emb结果不一致

Open
#6,364 21 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

GPU:H20
vLLM版本:0.10.1

swift部署的Emb

  • swift 启动部署的命令
swift deploy --model Alibaba-NLP/gme-Qwen2-VL-2B-Instruct --served_model_name gme_emb --vllm_max_model_len 2048 --port 8092 --host 0.0.0.0 --infer_backend vllm --torch_dtype float16 --task_type embedding --vllm_max_num_seqs 256 --max_length 2048 --vllm_gpu_memory_utilization 0.9
  • 服务测试Curl
curl http://localhost:8092/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
      "model": "gme_emb",
      "input": "苹果手机"
  }'

vLLM部署的Emb

  • vLLM启动部署的命令
vllm serve Alibaba-NLP/gme-Qwen2-VL-2B-Instruct --served-model-name gme_emb --port 9092 --max-model-len 2048 --host 0.0.0.0 --dtype float16 --trust-remote-code --task embedding --enable-prefix-caching --gpu-memory-utilization 0.9 --seed 42
  • 服务测试Curl
curl http://localhost:9092/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
      "model": "gme_emb",
      "input": "苹果手机"
  }'

结果差异

  • swift部署服务返回Emb:[3.931640625,-1.197265625,1.5869140625,-3.466796875,7.328125,-4.58984375,-1.0654296875, …]
  • vllm部署服务返回Emb:[-0.0229949951171875,-0.0164642333984375,0.042449951171875,0.0245513916015625,-0.017364501953125, …]
  • 两个返回向量cos相似度仅为0.8左右,明显不是归一化的原因【看前两个元素正负符号对不上】
  • 且有个细节,vLLM部署的服务返回usage信息中,"prompt_tokens":2; swift部署服务返回usage信息中,"prompt_tokens":22;

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the two deployments using the documented swift deploy and vLLM serve commands, then send the same /v1/embeddings request to both endpoints. Compare the returned vectors, cosine similarity, and prompt_tokens values, focusing on the differing token counts and embedding scale. Done means the cause of the discrepancy is identified and the services produce consistent embedding results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.