[PyTorch backend][Qwen3] embed_tokens ignores allreduce_strategy and falls back to AUTO

Summary

On the PyTorch backend, Qwen3 does not propagate model_config.allreduce_strategy into the TP all-reduce used by embed_tokens.
As a result, even when users explicitly configure allreduce_strategy: NCCL or NCCL_SYMMETRIC, the embedding path silently falls back to Linear(... allreduce_strategy=AUTO).

In our case, this also surfaced as a startup failure on TP=2 with:

RuntimeError: CUDA Runtime API error: <cudaError_t.cudaErrorInvalidDevice: 101>

coming from custom allreduce workspace initialization.

Expected behavior

allreduce_strategy should be applied consistently to all TP all-reduce sites in Qwen3, including the embedding / LMHead path.
If the user sets NCCL (or another explicit strategy), Qwen3 should not silently use AUTO for embed_tokens.

Actual behavior

In Qwen3Model, embed_tokens is constructed without passing model_config.allreduce_strategy:

self.embed_tokens = Embedding(
    config.pretrained_config.vocab_size,
    config.pretrained_config.hidden_size,
    dtype=config.pretrained_config.torch_dtype,
    mapping=config.mapping,
    tensor_parallel_mode=TensorParallelMode.COLUMN,
    gather_output=True,
)

However, Embedding inherits from LMHead -> Linear, and Linear defaults to:

allreduce_strategy: AllReduceStrategy = AllReduceStrategy.AUTO

So for TP>1, the embedding path ends up using AUTO instead of the user-configured strategy.

Why this looks like a bug

Other models already forward model_config.allreduce_strategy explicitly into TP modules.
For example, in modeling_cohere2.py the Linear(...) construction passes:

allreduce_strategy=model_config.allreduce_strategy

But modeling_qwen3.py currently has no such propagation for embed_tokens.

Relevant code paths

• tensorrt_llm/_torch/model_config.py
  • allreduce_strategy: AllReduceStrategy = AllReduceStrategy.AUTO
• tensorrt_llm/_torch/models/modeling_qwen3.py
  • self.embed_tokens = Embedding(...) without allreduce_strategy=...
• tensorrt_llm/_torch/modules/embedding.py
  • for TP>1 and column mode, output = self.all_reduce(output)
• tensorrt_llm/_torch/modules/linear.py
  • default allreduce_strategy=AllReduceStrategy.AUTO

Observed failure path

When this falls back to AUTO, initialization can enter the custom allreduce workspace path via:

• tensorrt_llm/_torch/distributed/ops.py
  • get_allreduce_workspace(...)
• tensorrt_llm/plugin/plugin.py
  • allocate_allreduce_fusion_workspace(...)

In our environment this eventually failed during TP=2 startup with:

RuntimeError: CUDA Runtime API error: <cudaError_t.cudaErrorInvalidDevice: 101>

Minimal reproduction idea

• Backend: pytorch
• Model: Qwen3 family (we observed this on Qwen3-32B-FP8)
• TP: 2
• Explicit config:

allreduce_strategy: NCCL

• Start trtllm-serve
• Observe that Qwen3 embedding all-reduce still behaves as if strategy were AUTO

Suggested fix

Propagate model_config.allreduce_strategy through the Qwen3 embedding path, e.g. by adding an allreduce_strategy parameter to Embedding / LMHead where needed and passing it from modeling_qwen3.py.