ROCm / ROCm/FastFlowLM

Feature Request / Question: Support for Qwen3-Embedding-8B (.q4nx) on XDNA2

Open
#491 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

model request
Dominant language
C++
Stars
1.9k
Forks
152
Avg merge
4h 14m
Merged PRs (30d)
11

Description

Hey team,

I'm looking to run local RAG workflows directly on the NPU (currently deploying on a Strix Halo APU) and would heavily prefer to use Qwen3-Embedding-8B to offload the embedding process.

I see that the core Qwen 3 and Qwen 3.5 architectures are already mapped and supported for text generation and vision models in the registry. However, I noticed there isn't an officially published .q4nx file for the Qwen3-Embedding-8B variant yet.

I have a couple of questions regarding the path forward:

Official Support: Are there plans to officially publish and maintain the .q4nx for this embedding model in the registry so it can be natively pulled via flm pull?

Custom Compilation: If I were to compile it myself, are there any known gotchas for this specific architecture? note its an embedding model

If there is a reference script or specific flm-build configuration you recommend for adapting the feature-extraction graph of Qwen 3, that would be incredibly helpful.

Thanks for the great work on the engine!

Johnny

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Qwen 3 registry mappings and the flm-build configuration or feature-extraction graph referenced in the issue; verify how embedding models differ from text-generation models on XDNA2. Check whether a Qwen3-Embedding-8B .q4nx artifact can be built and pulled via flm pull, and document the required configuration or support decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai-infra-agents
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.