NVIDIA / NVIDIA/TensorRT-LLM

[Feature]: Add Qwen3-Embedding model support (0.6B/4B/8B) for LLM API encode_only mode

Open
#14,715 1 comment 0 reactions 1 assignee View on GitHub

@laikhtewari is already working on this.

Since May 29, 2026.

feature request LLM API Model customization
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

🚀 The feature, motivation and pitch

Native support for the Qwen3-Embedding model family (0.6B, 4B, 8B) for use with the LLM API's encode_only mode.

Problem

The Qwen3 embedding models don't work correctly out of the box with encode_only. TensorRT-LLM returns logits (post lm_head), whereas the correct output should be the last prompt token's hidden state before the lm_head (model card's transformers usage)

Proposed support

Add a new architecture/model class, e.g. Qwen3ForEmbedding, for use with LLM(..., encode_only=True).

The model class would:

  • Reuse the existing Qwen3 transformer backbone without applying lm_head.
  • Return last prompt token's hidden states from LLM.encode(...) instead of logits.
  • Leave normalization and MRL/custom-dimension slicing to the caller or serving layer.
Alternatives

No response

Additional context

Models: Qwen3-Embedding-0.6B, Qwen3-Embedding-4B, Qwen3-Embedding-8B

I'm happy to contribute a PR for this feature

Before submitting a new issue...
  • Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.