推理时报错RuntimeError: CUDA error: device-side assert triggered

使用的模型是003的plugin int4模型，显卡是3090，报错时显存显示占用了12.5G。
具体报错信息如下：
/opt/conda/conda-bld/pytorch_1670525552843/work/aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [272,0,0], thread: [127,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
Exception in thread Thread-7 (generate):
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/opt/conda/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 1485, in generate
return self.sample(
File "/opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py", line 2524, in sample
outputs = self(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/moss-003-sft-plugin-int4-g/modeling_moss.py", line 674, in forward
transformer_outputs = self.transformer(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/moss-003-sft-plugin-int4-g/modeling_moss.py", line 545, in forward
outputs = block(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/moss-003-sft-plugin-int4-g/modeling_moss.py", line 270, in forward
attn_outputs = self.attn(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/moss-003-sft-plugin-int4-g/modeling_moss.py", line 192, in forward
k_rot = apply_rotary_pos_emb(k_rot, sin, cos)
File "/root/.cache/huggingface/modules/transformers_modules/moss-003-sft-plugin-int4-g/modeling_moss.py", line 56, in apply_rotary_pos_emb
sin = torch.repeat_interleave(sin[:, :, None, :], 2, 3)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Start with the reported forward path in modeling_moss.py, especially apply_rotary_pos_emb at line 56 and the calls through the attention and transformer blocks. Reproduce the generation failure with CUDA_LAUNCH_BLOCKING=1 and the reported MOSS 003 plugin int4 model on a 3090. Done means identifying and fixing the out-of-bounds indexing cause, with generation completing successfully.