AveragePool fails at the exact batch boundary N=65536 on RTX 5090

Description

A valid ONNX AveragePool builds and runs when the input batch dimension is
N=65535, but engine building fails at the exact boundary N=65536.

Input: [N, 1, 1, 512]
Output: [N, 1, 1, 256]
Kernel: [1, 2], stride: [1, 2], padding: 0

Latest TensorRT 11.1 reports:

[Autotuner]: No valid tactics to print (all tactics failed)
MyelinCheckException: CHECK(sorted_ids.size() > 0) failed. Must have costs
Could not find any implementation node {ForeignNode[node_avg_pool2d]}

TensorRT 10.13.3.9 also fails at N=65536, reporting
Cask Pooling Runner Execute Failure. Both versions succeed at N=65535.

Environment

TensorRT Version: 11.1.0.106 (tensorrt-cu12); also reproduced with 10.13.3.9.post1

NVIDIA GPU: NVIDIA GeForce RTX 5090

NVIDIA Driver Version: 575.64.03

CUDA Version: 12.9 / CUDA runtime 12.9.79

CUDNN Version: 9.10.2

Operating System: Ubuntu 24.04

Python Version (if applicable): 3.10.19

PyTorch Version (if applicable): 2.8.0+cu129

Baremetal or Container (if so, version): Baremetal

Steps To Reproduce

Commands or scripts:

  import time
  import numpy as np
  import onnx
  import tensorrt as trt
  from onnx import TensorProto, helper


  def create_model(n):
      path = f"avgpool_n{n}.onnx"
      node = helper.make_node(
          "AveragePool", ["input"], ["output"],
          name="node_avg_pool2d",
          kernel_shape=[1, 2], strides=[1, 2],
          pads=[0, 0, 0, 0],
          ceil_mode=0, count_include_pad=1,
      )
      graph = helper.make_graph(
          [node], f"avgpool_n{n}",
          [helper.make_tensor_value_info(
              "input", TensorProto.FLOAT, [n, 1, 1, 512])],
          [helper.make_tensor_value_info(
              "output", TensorProto.FLOAT, [n, 1, 1, 256])],
      )
      model = helper.make_model(
          graph, opset_imports=[helper.make_opsetid("", 18)],
          ir_version=10,
      )
      onnx.checker.check_model(model)
      onnx.save(model, path)
      return path


  def build(path):
      logger = trt.Logger(trt.Logger.ERROR)
      builder = trt.Builder(logger)
      network = builder.create_network(0)
      parser = trt.OnnxParser(network, logger)
      assert parser.parse_from_file(path), [
          str(parser.get_error(i)) for i in range(parser.num_errors)
      ]
      config = builder.create_builder_config()
      config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30)
      start = time.time()
      plan = builder.build_serialized_network(network, config)
      return plan is not None, time.time() - start


  print("TensorRT:", trt.__version__)
  for n in (65535, 65536):
      success, elapsed = build(create_model(n))
      print(n, "SUCCESS" if success else "FAILED", f"{elapsed:.3f}s")

Observed output:

TensorRT: 11.1.0.106
65535 SUCCESS 1.247s
65536 FAILED 13.870s

Full TensorRT 11.1 error:

[Autotuner]: No valid tactics to print (all tactics failed)
MyelinCheckException: autotuner.cpp:2899:
CHECK(sorted_ids.size() > 0) failed. Must have costs

Error Code: 9: Skipping tactic 0x0000000000000000 due exception
MyelinCheckException

IBuilder::buildSerializedNetwork: Error Code 10: Internal Error
Could not find any implementation node {ForeignNode[node_avg_pool2d]}.

Have you tried the latest release?:

The issue reproduces with the latest available tensorrt-cu12 release, 11.1.0.106.
It was also reproduced with TensorRT 10.13.3.9.post1.

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

• PyTorch native CUDA torch.nn.functional.avg_pool2d succeeds even with N=196608.
• ONNX Runtime CPU EP succeeds.
• ONNX Runtime CUDA EP fails at the same N=65536 boundary because its pooling path returns CUDNN_STATUS_NOT_SUPPORTED.
• Replacing the operation with the mathematically equivalent Reshape + ReduceMean graph succeeds on the same GPU and produces identical results.

Start with the provided Python reproducer and compare TensorRT builds for N=65535 and N=65536 using the AveragePool model. Check the ONNX parsing and serialized-network build path, then verify that the exact boundary succeeds while preserving the reported output shape and behavior.