pytorch / pytorch/TensorRT

🐛 [Bug] mul emits ElementWiseOperation.PROD for bool × bool, which TensorRT rejects (AND is supported)

Open
#4,605 0 comments 0 reactions 1 assignee View on GitHub

@micwill755 is already working on this.

Since Sep 17, 2026.

bug
Dominant language
Python
Stars
3k
Forks
410
Avg merge
3d 18h
Merged PRs (30d)
78

Description

Bug Description

Torch defines bool * bool as logical and. The mul converter lowers every
aten.mul.Tensor to trt.ElementWiseOperation.PROD regardless of operand dtype:

# py/torch_tensorrt/dynamo/conversion/impl/elementwise/ops.py :: mul
def mul(
    ctx: ConversionContext,
    target: Target,
    source_ir: Optional[SourceIR],
    name: str,
    lhs_val: Union[TRTTensor, int, float],
    rhs_val: Union[TRTTensor, int, float],
) -> TRTTensor:
    return convert_binary_elementwise(
        ctx,
        target,
        source_ir,
        name,
        trt.ElementWiseOperation.PROD,
        lhs_val,
        rhs_val,
    )

convert_binary_elementwise then computes the promoted type for the two operands, and for two
Bool operands that is Bool (this build uses torch.result_type; the 2.11.0 source used
torch.promote_types -- both give torch.bool), so no cast is inserted and TensorRT is handed
PROD on two Bool tensors, which it rejects at build time:

[ELEMENTWISE]-[aten_ops.mul.Tensor]-[mul]: ElementWiseOperation PROD requires inputs with type
kFLOAT, kHALF, kBF16, kFP8, or kINT8, kINT32, or kINT64. But type is Bool.

TensorRT is perfectly willing to run an elementwise op on Bool -- AND is supported -- and the
control in the reproducer proves it: torch.logical_and(a, b) on the same two bool inputs
compiles and runs, and the TensorRT log shows the Bool layer being built:

Layer(Myelin): {ForeignNode[[ELEMENTWISE]-[aten_ops.logical_and.default]-[logical_and]]}, Tactic: 0x0000000000000000, arg0_1 (Bool[4,8]), arg1_1 (Bool[4,8]) -> output0 (Bool[4,8])

So mul needs to emit AND when both operands are Bool (or cast them to an integer type),
and the case would work.

The diagnostics for this are the second half of the problem, and are worth fixing
separately.
The identification above rests entirely on the TensorRT log text. The Python
exception carries none of it: it is a bare AssertionError with an empty message, from
assert cuda_engine in _TRTInterpreter.run(). Note that the sibling branch a few lines
earlier does the right thing:

# py/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py :: run
        if serialized_engine is None:
            raise RuntimeError(
                "TensorRT build_serialized_network returned None; engine build failed."
            )
        ...
    else:
        cuda_engine = self.builder.build_engine_with_config(
            self.ctx.net, builder_config
        )
    assert cuda_engine

A user who has not attached a handler to the torch_tensorrt [TensorRT Conversion Context]
logger sees only BackendCompilerFailed: backend='tensorrt' raised: AssertionError: with no
message, no layer name, and no dtype -- nothing that points at mul, at Bool, or at
ElementWiseOperation. The reproducer has to install its own logging handler to recover the
cause. That is why the script captures that logger.

To Reproduce

Steps to reproduce the behavior:

docker run --rm --gpus all --ipc=host -v "$PWD":/w -w /w \
        nvcr.io/nvidia/pytorch:26.07-py3 python repro.py

repro.py

import logging
import traceback

import torch

import torch_tensorrt  # noqa: F401  # registers the "tensorrt" torch.compile backend

COMPILE_OPTIONS = {"pass_through_build_failures": True, "min_block_size": 1}
TRT_LOGGER_NAME = "torch_tensorrt [TensorRT Conversion Context]"


class MulBoolBool(torch.nn.Module):
    """`bool * bool` -- torch computes logical and, the converter emits PROD."""

    def forward(self, a: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
        return a * b


class LogicalAndBoolBool(torch.nn.Module):
    """Control: the same computation spelled so the converter emits AND."""

    def forward(self, a: torch.Tensor, b: torch.Tensor) -> torch.Tensor:
        return torch.logical_and(a, b)


class _RecordingHandler(logging.Handler):
    def __init__(self) -> None:
        super().__init__(level=logging.DEBUG)
        self.messages: list[str] = []

    def emit(self, record: logging.LogRecord) -> None:
        self.messages.append(record.getMessage())


def compile_and_run(
    model: torch.nn.Module, args: tuple[torch.Tensor, ...]
) -> tuple[BaseException | None, list[str]]:
    """Compiles with the TensorRT backend. Returns (exception or None, TensorRT log messages)."""
    torch._dynamo.reset()
    handler = _RecordingHandler()
    trt_logger = logging.getLogger(TRT_LOGGER_NAME)
    trt_logger.setLevel(logging.DEBUG)
    trt_logger.addHandler(handler)
    optimized = torch.compile(model, backend="tensorrt", options=COMPILE_OPTIONS)
    try:
        optimized(*args)
        return None, handler.messages
    except BaseException as exc:  # noqa: BLE001  # a bug demonstration: report, do not handle
        traceback.print_exc()
        return exc, handler.messages
    finally:
        trt_logger.removeHandler(handler)


def main() -> int:
    print(f"torch {torch.__version__}")
    print(f"torch_tensorrt {torch_tensorrt.__version__}")

    a = torch.rand(4, 8, device="cuda") > 0.5
    b = torch.rand(4, 8, device="cuda") > 0.5

    print("\n=== control: torch.logical_and(a, b) on bool inputs (expects AND) ===")
    control_exc, _ = compile_and_run(LogicalAndBoolBool().cuda().eval(), (a, b))
    control_ok = control_exc is None
    print(f"control compiled: {control_ok}")

    print("\n=== bug case: a * b on bool inputs (emits PROD) ===")
    bug_exc, messages = compile_and_run(MulBoolBool().cuda().eval(), (a, b))
    prod_bool_errors = [m for m in messages if "PROD" in m and "Bool" in m]
    for message in prod_bool_errors:
        print(f"TensorRT log: {message}")
    if bug_exc is None:
        print("bug case compiled cleanly -- the bug did not reproduce")
        reproduced = False
    else:
        print(f"exception: {type(bug_exc).__name__}: {bug_exc}")
        print(f"TensorRT rejected PROD on Bool: {bool(prod_bool_errors)}")
        reproduced = bool(prod_bool_errors)

    print(f"\ncontrol compiles: {control_ok}")
    print(f"reproduced: {reproduced}")
    return 0 if (reproduced and control_ok) else 1


if __name__ == "__main__":
    raise SystemExit(main())

repro.py

Backtrace

torch 2.13.0a0+9186a08b2c.nv26.07
torch_tensorrt 2.14.0a0

=== control: torch.logical_and(a, b) on bool inputs (expects AND) ===
Layer(Myelin): {ForeignNode[[ELEMENTWISE]-[aten_ops.logical_and.default]-[logical_and]]}, Tactic: 0x0000000000000000, arg0_1 (Bool[4,8]), arg1_1 (Bool[4,8]) -> output0 (Bool[4,8])
control compiled: True

=== bug case: a * b on bool inputs (emits PROD) ===
ERROR:torch_tensorrt [TensorRT Conversion Context]:ITensor::getDimensions: Error Code 4: API Usage Error ([ELEMENTWISE]-[aten_ops.mul.Tensor]-[mul]: ElementWiseOperation PROD requires inputs with type kFLOAT, kHALF, kBF16, kFP8, or kINT8, kINT32, or kINT64. But type is Bool. In validateTypes at /_src/optimizer/common/nodes/elementWiseNode.cpp:69)
ERROR:torch_tensorrt [TensorRT Conversion Context]:ITensor::getDimensions: Error Code 4: API Usage Error (Output shape can not be computed for node [ELEMENTWISE]-[aten_ops.mul.Tensor]-[mul]. In needTypeAndDimensions at /_src/optimizer/shapeof/graphShapeAnalyzer.cpp:2986)
ERROR:torch_tensorrt [TensorRT Conversion Context]:IBuilder::buildEngineWithConfig: Error Code 4: API Usage Error ([ELEMENTWISE]-[aten_ops.mul.Tensor]-[mul]: ElementWiseOperation PROD requires inputs with type kFLOAT, kHALF, kBF16, kFP8, or kINT8, kINT32, or kINT64. But type is Bool. In validateTypes at /_src/optimizer/common/nodes/elementWiseNode.cpp:69)
CRITICAL:torch_tensorrt.dynamo.backend.backends:Halting compilation on build failure since pass_through_build_failures was specified as True. To return the default Torch implementation and avoid halting compilation on engine build failures, specify pass_through_build_failures=False.
Traceback (most recent call last):
  File "/w/repro.py", line 78, in compile_and_run
    optimized(*args)
[... dynamo frames elided ...]
  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/_compiler.py", line 1141, in compile_module
    trt_module = convert_module(
                 ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/conversion/_conversion.py", line 347, in convert_module
    serialized_interpreter_result = interpret_module_to_result(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/conversion/_conversion.py", line 280, in interpret_module_to_result
    interpreter_result = interpreter.run()
                         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch_tensorrt/dynamo/conversion/_TRTInterpreter.py", line 653, in run
    assert cuda_engine
           ^^^^^^^^^^^
torch._dynamo.exc.BackendCompilerFailed: backend='tensorrt' raised:
AssertionError: 

TensorRT log: ITensor::getDimensions: Error Code 4: API Usage Error ([ELEMENTWISE]-[aten_ops.mul.Tensor]-[mul]: ElementWiseOperation PROD requires inputs with type kFLOAT, kHALF, kBF16, kFP8, or kINT8, kINT32, or kINT64. But type is Bool. In validateTypes at /_src/optimizer/common/nodes/elementWiseNode.cpp:69)
TensorRT log: IBuilder::buildEngineWithConfig: Error Code 4: API Usage Error ([ELEMENTWISE]-[aten_ops.mul.Tensor]-[mul]: ElementWiseOperation PROD requires inputs with type kFLOAT, kHALF, kBF16, kFP8, or kINT8, kINT32, or kINT64. But type is Bool. In validateTypes at /_src/optimizer/common/nodes/elementWiseNode.cpp:69)
exception: BackendCompilerFailed: backend='tensorrt' raised:
AssertionError: 

TensorRT rejected PROD on Bool: True

control compiles: True
reproduced: True

Expected behavior

a * b on two bool tensors should compile and produce the same result as
torch.logical_and(a, b). The case is expressible in TensorRT, so this is a missing lowering
rather than a capability gap: mul should emit trt.ElementWiseOperation.AND when both
promoted operands are Bool (or, equivalently, cast both to an integer type and keep PROD).
Either way bool * bool should not need a source change in the model.

Second, separate request: an engine build failure should raise an exception that says what
failed.
Today the failure arrives as a bare AssertionError with an empty message from
assert cuda_engine, and everything diagnostic lives in the TensorRT logger output. Replacing
that assert with a RuntimeError carrying the builder's error -- matching what the
build_serialized_network branch a few lines above already does -- would make this class of
failure self-diagnosing. As it stands, a user hitting this in a large model has no indication
which node or which dtype is at fault.

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

  • Pytorch NGC Container : 26.07-py3

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.