T5 Onnx to TensorRT conversion Error: Internal Error (encoder_hidden_states: for dimension number 2 in profile 0 does not match network definition (got min=2048, opt=2048, max=2048), expected min=opt=max=1024).)

Description

Followed https://github.com/NVIDIA/TensorRT/blob/main/demo/HuggingFace/notebooks/t5.ipynb to convert a custom fine-tuned T5-large model into TensorRT engine.

Points to note about T5-large model:
-> Fine-tuned with a max_sequence_length of 2048

Error obtained while running the conversion script with a max_sequence_length of 2048:
"Error Code 4: Internal Error (encoder_hidden_states: for dimension number 2 in profile 0 does not match network definition (got min=2048, opt=2048, max=2048), expected min=opt=max=1024).)"

Points to note about the conversion script:
-> Works as expected when I use a max_sequence_length of 1024

Environment

TensorRT Version: 8.6.0.12
NVIDIA GPU: T4-16GB
NVIDIA Driver Version: 515.65.01
CUDA Version: 12.0
CUDNN Version: 8.08
Operating System: Ubuntu 20.04.5 LTS
Python Version (if applicable): 3.8.10
Tensorflow Version (if applicable):
PyTorch Version (if applicable): 1.10.2+cu113
Baremetal or Container (if so, version):

Relevant Files

Attached the script I am using for converting my custom fine-tuned t5-large model into TensorRT format. This is based off of https://github.com/NVIDIA/TensorRT/blob/main/demo/HuggingFace/notebooks/t5.ipynb.
t5_inference_with_tensorrt.py.zip

Steps To Reproduce

1. I create a Sagemaker notebook instance with 16 GB GPU memory
2. Within the instance, I clone the TensorRT repo (https://github.com/NVIDIA/TensorRT.git) and run step 1 in "Downloading TensorRT Build" section
3. I build the TensorRT OSS container with "./docker/build.sh --file docker/ubuntu-20.04.Dockerfile --tag tensorrt-ubuntu20.04-cuda12.0"
4. Then, I launch the container with "./docker/launch.sh --tag tensorrt-ubuntu20.04-cuda12.0 --gpus all"
5. Then, I install the requirements from TensorRT/demo/HuggingFace directory with "pip3 install -r requirements.txt"
6. I then copy the model files from the instance into the docker container
7. I make the following changes to the "TensorRT/demo/HuggingFace/T5/T5ModelConfig.py" file:
   a) MAX_SEQUENCE_LENGTH = {
   TARGET_MODELS[0]: 512,
   TARGET_MODELS[1]: 768,
   TARGET_MODELS[2]: 2048, # 1024 -> 2048
   TARGET_MODELS[3]: 1024,
   TARGET_MODELS[4]: 1024,
   }
   b) MAX_OUTPUT_LENGTH = {
   TARGET_MODELS[0]: 512,
   TARGET_MODELS[1]: 768,
   TARGET_MODELS[2]: 512, # 1024 -> 512
   TARGET_MODELS[3]: 1024,
   TARGET_MODELS[4]: 1024,
   }
8. Then I step into an ipython environment and run the steps in the attached script (t5_inference_with_tensorrt.py)
9. I get the following error at the triton decoder engine creation step: "t5_trt_decoder_engine = T5DecoderONNXFile(os.path.join(onnx_model_path, decoder_onnx_model_fpath), metadata).as_trt_engine(
   decoder_engine_name,
   profiles=[decoder_profile],
   preview_features=preview_features)"

Error Message:

[E] 4: [network.cpp::validate::3084] Error Code 4: Internal Error (encoder_hidden_states: for dimension number 2 in profile 0 does not match network definition (got min=2048, opt=2048, max=2048), expected min=opt=max=1024).)
[!] Invalid Engine. Please ensure the engine was built correctly

PolygraphyException Traceback (most recent call last)
Cell In[157], line 1
----> 1 trt_engine = engine_from_network(network_definition, config=trt_inference_config)

File :3, in engine_from_network(network, config, save_timing_cache)

File /usr/local/lib/python3.8/dist-packages/polygraphy/backend/base/loader.py:42, in BaseLoader.call(self, *args, **kwargs)
36 """
37 Invokes the loader by forwarding arguments to call_impl.
38
39 Note: call_impl should not be called directly - use this function instead.
40 """
41 doc = self.call_impl.doc
---> 42 return self.call_impl(*args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/polygraphy/backend/trt/loader.py:530, in EngineFromNetwork.call_impl(self)
524 """
525 Returns:
526 trt.ICudaEngine: The engine that was created.
527 """
528 # We do not invoke super().call_impl here because we would otherwise be responsible
529 # for freeing it's return values.
--> 530 return engine_from_bytes(super().call_impl)

File :3, in engine_from_bytes(serialized_engine)

File /usr/local/lib/python3.8/dist-packages/polygraphy/backend/base/loader.py:42, in BaseLoader.call(self, *args, **kwargs)
36 """
37 Invokes the loader by forwarding arguments to call_impl.
38
39 Note: call_impl should not be called directly - use this function instead.
40 """
41 doc = self.call_impl.doc
---> 42 return self.call_impl(*args, **kwargs)

File /usr/local/lib/python3.8/dist-packages/polygraphy/backend/trt/loader.py:554, in EngineFromBytes.call_impl(self)
549 def call_impl(self):
550 """
551 Returns:
552 trt.ICudaEngine: The deserialized engine.
553 """
--> 554 buffer, owns_buffer = util.invoke_if_callable(self._serialized_engine)
556 trt.init_libnvinfer_plugins(trt_util.get_trt_logger(), "")
557 with contextlib.ExitStack() as stack, trt.Runtime(trt_util.get_trt_logger()) as runtime:

File /usr/local/lib/python3.8/dist-packages/polygraphy/util/util.py:661, in invoke_if_callable(func,*args, **kwargs)
656 """
657 Attempts to invoke a function with arguments. If func is not callable, then returns func
658 The second return value of this function indicates whether the argument was a callable.
659 """
660 if callable(func):
--> 661 ret = func(*args, **kwargs)
662 return ret, True
663 return func, False

File /usr/local/lib/python3.8/dist-packages/polygraphy/backend/trt/loader.py:488, in EngineBytesFromNetwork.call_impl(self)
485 end_time = time.time()
487 if not engine_bytes:
--> 488 G_LOGGER.critical("Invalid Engine. Please ensure the engine was built correctly")
490 G_LOGGER.finish(f"Finished engine building in {end_time - start_time:.3f} seconds")
492 if self.timing_cache_path:

File /usr/local/lib/python3.8/dist-packages/polygraphy/logger/logger.py:597, in Logger.critical(self, message)
594 self.log(message, Logger.CRITICAL, stack_depth=3)
595 from polygraphy.exception import PolygraphyException
--> 597 raise PolygraphyException(message) from None

PolygraphyException: Invalid Engine. Please ensure the engine was built correctly

** My understanding of the error **:
The network definition is created off of the onnx model and that has 1024 as the third dimension (-1, -1, 1024) whereas I specify 2048 as the sequence length in "profile creation" which is fed as input to the onnx->Trt engine creation step.
What I don't understand is that why is onnx model conversion step not considering this fact? Am I required to change anything in the HF model's config?