How to strictly limiting the maximum GPU memory usage and clear GPU memory cache?

My use case scenario is deploying model inference services in the cloud, utilizing GPU virtualization technology to split one GPU into multiple instances. Each instance runs a model, and since one card has a total of about 22GB of available memory, I divided it into 10 instances, with each instance allocated 2GB of memory.
From the conversion logs of trtexec, I noticed that executing each model with TensorRT inference approximately consumes around 1.6GB of memory, so I thought allocating 2GB for each instance would be sufficient. However, during concurrent testing, it seems there was a memory overflow, which means the memory usage for each model inference exceeded 2GB.
When converting the ONNX model to a TensorRT model, I set workspace=2048, so it shouldn't exceed 2GB, right? Therefore, how can I ensure that the maximum memory usage for model inference does not exceed the workspace?
Additionally, I'm not sure if the increasing number of model inference calls led to a gradual increase in memory usage, ultimately resulting in an OOM error. So I considered clearing the cache after each inference. How to clear the GPU cache after the model has finished inference? Is it done using torch.cuda.empty_cache() or does TensorRT have its own related APIs?
I have posted a issue here, there are some log infomation.

Start by reviewing the linked TensorRT issue 4136 and the trtexec conversion logs mentioned here. Compare the reported workspace setting with the concurrent inference behavior, and check the roles of torch.cuda.empty_cache() and TensorRT APIs. Done means establishing the supported memory limits and cache-management behavior.