nextlevelbuilder / nextlevelbuilder/goclaw

vLLM + non-Matryoshka model: error when passing 'dimensions' field; request to add vLLM support

Open
#804 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

agent:github-maintain area:providers bug maintain:bug-confirmed maintain:triaged model-integration P2-medium
Dominant language
Go
Stars
3.6k
Forks
1.1k
Avg merge
3d 5h
Merged PRs (30d)
24

Description

Image

Issue Description

When using vLLM as the inference backend with a non-Matryoshka embedding model (i.e., a model that does not natively support flexible output dimensions), an error occurs because the dimensions field is passed upstream.

Root Cause

vLLM may be sending a dimensions parameter that is not supported by non-Matryoshka models, causing the inference to fail.

Proposed Solution

Do NOT pass the dimensions field upstream for non-Matryoshka models. Instead, always use the model's native output dimension.

If shorter vectors are needed for business purposes (e.g., storage efficiency or retrieval speed), the truncation/dimension reduction should be handled at the application level after obtaining the full vector from the model.

Additional Request

Please consider adding official vLLM support for embedding models in this project, especially for non-Matryoshka architectures.

Environment

vLLM version: vllm --version
0.19.0

Model used: [gemma-2b-embeddings]

app version: [latest:dev,windows-web-postgres]

Expected Behavior

Non-Matryoshka models should work with vLLM without errors.

The dimensions parameter should not be sent to models that don't support it.

Workaround (for now)

Users can manually override the request to remove the dimensions field, or switch to a different inference backend (e.g., Hugging Face Transformers).

Thank you for your great work! Looking forward to vLLM support.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are identified in the issue. Start by tracing the vLLM embedding request path and how the dimensions parameter is forwarded for non-Matryoshka models; done means those models work without errors while preserving the native output dimension.

Written by the indexing model from the issue text.

Assessment

Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.