[Feature]: Add Qwen3-Embedding model support (0.6B/4B/8B) for LLM API encode_only mode
@laikhtewari is already working on this.
Since May 29, 2026.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
🚀 The feature, motivation and pitch
Native support for the Qwen3-Embedding model family (0.6B, 4B, 8B) for use with the LLM API's encode_only mode.
Problem
The Qwen3 embedding models don't work correctly out of the box with encode_only. TensorRT-LLM returns logits (post lm_head), whereas the correct output should be the last prompt token's hidden state before the lm_head (model card's transformers usage)
Proposed support
Add a new architecture/model class, e.g. Qwen3ForEmbedding, for use with LLM(..., encode_only=True).
The model class would:
- Reuse the existing Qwen3 transformer backbone without applying lm_head.
- Return last prompt token's hidden states from
LLM.encode(...)instead of logits. - Leave normalization and MRL/custom-dimension slicing to the caller or serving layer.
Alternatives
No response
Additional context
Models: Qwen3-Embedding-0.6B, Qwen3-Embedding-4B, Qwen3-Embedding-8B
I'm happy to contribute a PR for this feature
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.