Integer overflow in slat decoding

Hi,

Thank you for the enormous efforts in open-sourcing this amazing project!

I'm testing the model on some in-the-wild images. Most images work fine, but on a small number of images there is the following error:

[Exception|implicit_gemm]feat=torch.Size([7342720, 192]),w=torch.Size([96, 3, 3, 3, 192]),pair=torch.Size([27, 7342720]),act=7342720,issubm=True,istrain=True

Upon further investigation it seems like there is an overflow occurring for the underlying spconv library:

/home/ruiningli/spconv/spconv/build/core_cc/src/cumm/conv/main/ConvMainUnitTest/ConvMainUnitTest_matmul_split_Ampere_f16f16f16_0.cu(294)                                                                                                             
int64_t(N) * int64_t(C) * tv::bit_size(algo_desp.dtype_a) / 8 < int_max assert faild. your data exceed int32 range. this will be fixed in cumm + nvrtc (spconv 2.2/2.3).

I'm wondering if anyone has encountered the same during training / inference, and how I might be able to get around this.

Many thanks!
Ruining

Start by reproducing the failure during slat decoding with an affected in-the-wild image, then trace the implicit_gemm call associated with the reported spconv assertion. Done means the affected images decode without the integer-overflow failure, or the supported workaround and limitation are documented.