NVIDIA / NVIDIA/TensorRT

TensorRT8.5.2 conversion failed when converting groups convolutions

Open
#3,273 11 comments 0 reactions 1 assignee View on GitHub

@zerollzeng is already working on this.

Since Aug 31, 2023.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

It will report an error when converting a model with group convolution using TensorRT8.5.2.

Environment

TensorRT Version:
8.5.2.2

Relevant Files

The error message is as follows:

[08/29/2023-21:40:17] [TRT] [E] Conv_221:kernel weights has count 294912 but 147456 was expected
[08/29/2023-21:40:17] [TRT] [E] Conv_221: count of 294912 weights in kernel, but kernel dimensions (3,3) with 128 input channels, 256 output channels and 2 groups were specified. Expected Weights count is 128 * 3*3 * 256 / 2 = 147456
[08/29/2023-21:40:17] [TRT] [E] [convolutionNode.cpp::computeOutputExtents::57] Error Code 4: Internal Error (Conv_221: number of kernel weights does not match tensor dimensions)
[08/29/2023-21:40:17] [TRT] [E] ModelImporter.cpp:726: While parsing node number 221 [Conv -> "input.487"]:
[08/29/2023-21:40:17] [TRT] [E] ModelImporter.cpp:727: --- Begin node ---
[08/29/2023-21:40:17] [TRT] [E] ModelImporter.cpp:728: input: "input.483"
input: "bbox_head.reg_convs.1.conv.weight"
input: "bbox_head.reg_convs.1.conv.bias"
output: "input.487"
name: "Conv_221"
op_type: "Conv"
attribute {
  name: "dilations"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group"
  i: 2
  type: INT
}
attribute {
  name: "kernel_shape"
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "pads"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides"
  ints: 1
  ints: 1
  type: INTS
}

[08/29/2023-21:40:17] [TRT] [E] ModelImporter.cpp:729: --- End node ---
[08/29/2023-21:40:17] [TRT] [E] ModelImporter.cpp:731: ERROR: ModelImporter.cpp:172 In function parseGraph:
[6] Invalid Node - Conv_221
Conv_221:kernel weights has count 294912 but 147456 was expected
Conv_221: count of 294912 weights in kernel, but kernel dimensions (3,3) with 128 input channels, 256 output channels and 2 groups were specified. Expected Weights count is 128 * 3*3 * 256 / 2 = 147456
[convolutionNode.cpp::computeOutputExtents::57] Error Code 4: Internal Error (Conv_221: number of kernel weights does not match tensor dimensions)

The error message display was caused by a mismatch between the weight and the actual convolution, but I was able to run normally on Pytorch using the same model and configuration file. The error message displays "but kernel dimensions (3,3) with 128 input channels, 256 output channels, and 2 groups were specified." However, in reality, the number of input channels for this convolution should be 256. The number of input channels I used when building the model is 256, which can be determined. Therefore, I suspect that grouping convolution caused an error in TensorRT calculation, as the model is actually continuous.

In addition, another issue was encountered while converting shuffleNet.

[08/29/2023-17:23:23] [TRT] [E] [convolutionNode.cpp::computeOutputExtents::54] Error Code 4: Internal Error (Conv_136: group count must divide input channel count)
[08/29/2023-17:23:23] [TRT] [E] ModelImporter.cpp:726: While parsing node number 136 [Conv -> "input.131"]:
[08/29/2023-17:23:23] [TRT] [E] ModelImporter.cpp:727: --- Begin node ---
[08/29/2023-17:23:23] [TRT] [E] ModelImporter.cpp:728: input: "input.123"
input: "onnx::Conv_2562"
input: "onnx::Conv_2563"
output: "input.131"
name: "Conv_136"
op_type: "Conv"
attribute {
  name: "dilations"
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group"
  i: 116
  type: INT
}
attribute {
  name: "kernel_shape"
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "pads"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides"
  ints: 2
  ints: 2
  type: INTS
}

[08/29/2023-17:23:23] [TRT] [E] ModelImporter.cpp:729: --- End node ---
[08/29/2023-17:23:23] [TRT] [E] ModelImporter.cpp:731: ERROR: ModelImporter.cpp:172 In function parseGraph:
[6] Invalid Node - Conv_136
[convolutionNode.cpp::computeOutputExtents::54] Error Code 4: Internal Error (Conv_136: group count must divide input channel count)

It is worth mentioning that both of these use the shuffle operation, which is the following code:

def channel_shuffle(x, groups):
   Channel Shuffle operation.

    This function enables cross-group information flow for multiple groups
    convolution layers.

    Args:
        x (Tensor): The input tensor.
        groups (int): The number of groups to divide the input tensor
            in the channel dimension.

    Returns:
        Tensor: The output tensor after channel shuffle operation.


    batch_size, num_channels, height, width = x.size()
    assert (num_channels % groups == 0), ('num_channels should be '
                                          'divisible by groups')
    channels_per_group = num_channels // groups

    x = x.view(batch_size, groups, channels_per_group, height, width)
    x = torch.transpose(x, 1, 2).contiguous()
    x = x.view(batch_size, -1, height, width)

    return x

Will this operation also affect the correct transformation of the TensorRT model?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.