NVIDIA / NVIDIA/TensorRT-LLM

[Bug]: construct_if_true in moe_gemm_tma_ws_launcher.inl fails to compile at C++20

Open
#19,063 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Customized kernels
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info
  • CPU architecture: x86_64
  • GPU: not required (compile-only)
  • TensorRT-LLM: main, commit 4476ee3eacc1639c28f7e3e845b67ec246f2e482
  • CUDA: 13.4, nvcc V13.4.59
  • Host compiler: gcc 13.3.0 (Ubuntu 24.04)
  • Container: nvcr.io/nvidia/cuda-dl-base:26.08-cuda13.4-devel-ubuntu24.04
Who can help?

Build / MoE kernels.

Description

This is a second, separate C++20 blocker found while working on #19059. It is
unrelated to the functional-cast problem fixed in #19060, so I am filing it on its
own as suggested there.

Building the grouped-GEMM instantiations at C++20 fails in
cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.inl:78:

moe_gemm_tma_ws_launcher.inl(78): error: a value of type "const float **" cannot be
used to initialize an entity of type "float"
          return ReturnType{std::forward<Args>(args)...};
                            ^
          detected during instantiation of
          "ReturnType tensorrt_llm::_v1::kernels::cutlass_kernels_oss::construct_if_true<FLAG, ReturnType, Args...>"

The helper is:

// Constructs an object with specific arguments only if flag is true
// This forces the if constexpr branch to properly pruned be when called from in non-template functions
template <bool FLAG, class ReturnType, class... Args>
ReturnType construct_if_true(Args&&... args)
{
    if constexpr (FLAG)
    {
        return ReturnType{std::forward<Args>(args)...};
    }
    else
    {
        return ReturnType{};
    }
}

Call sites are at lines 551, 565, 582, 590 (MainloopArguments) and 610
(EpilogueScalars).

Standard dependence

Same tree, same flags, only -std changed:

target C++17 C++20
build/.../cutlass_instantiations/gemm_grouped/100/cutlass_kernel_file_gemm_grouped_sm100_M128_BS_group0.generated.cu 0 errors 2 errors

In a full --cpp_only build at C++20 (with #19060 applied so the fp8 casts no longer
interfere), this is the only file left with errors: 24 diagnostics, all from line 78,
which is the same line instantiated 24 times. The build otherwise reaches 95%.

Why C++20 comes up at all

requirements.txt:29 pins torch>=2.12.0a0, and torch 2.12 declares
CXX_STANDARD 20 in share/cmake/Torch/TorchConfig.cmake:158. At C++17 nvcc cannot
compile the torch headers at all, so raising CMAKE_CUDA_STANDARD is the only way I
found to get a build going outside the prebuilt container. Details are in #19059.

What I did not investigate

I have not traced which MainloopArguments / EpilogueScalars member receives the
const float **, so I do not know whether the right fix is in the helper, in the call
sites, or in the argument structs. I am not proposing a patch for this one, because
getting the argument mapping wrong here would silently produce incorrect results
rather than a compile error.

Reporting it so the C++20 picture is complete alongside #19059 and #19060.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.inl:78 and inspect the MainloopArguments and EpilogueScalars call sites at lines 551, 565, 582, 590, and 610. Trace which member receives the const float** during the generated grouped-GEMM C++20 instantiation, then verify the resolved behavior with the cited cutlass_instantiations target and a full --cpp_only build.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, pytorch
Domain
backend, build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.