nextlevelbuilder / nextlevelbuilder/goclaw
vLLM + non-Matryoshka model: error when passing 'dimensions' field; request to add vLLM support
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.1k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 24
Description
Issue Description
When using vLLM as the inference backend with a non-Matryoshka embedding model (i.e., a model that does not natively support flexible output dimensions), an error occurs because the dimensions field is passed upstream.
Root Cause
vLLM may be sending a dimensions parameter that is not supported by non-Matryoshka models, causing the inference to fail.
Proposed Solution
Do NOT pass the dimensions field upstream for non-Matryoshka models. Instead, always use the model's native output dimension.
If shorter vectors are needed for business purposes (e.g., storage efficiency or retrieval speed), the truncation/dimension reduction should be handled at the application level after obtaining the full vector from the model.
Additional Request
Please consider adding official vLLM support for embedding models in this project, especially for non-Matryoshka architectures.
Environment
vLLM version: vllm --version
0.19.0
Model used: [gemma-2b-embeddings]
app version: [latest:dev,windows-web-postgres]
Expected Behavior
Non-Matryoshka models should work with vLLM without errors.
The dimensions parameter should not be sent to models that don't support it.
Workaround (for now)
Users can manually override the request to remove the dimensions field, or switch to a different inference backend (e.g., Hugging Face Transformers).
Thank you for your great work! Looking forward to vLLM support.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are identified in the issue. Start by tracing the vLLM embedding request path and how the dimensions parameter is forwarded for non-Matryoshka models; done means those models work without errors while preserving the native output dimension.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100