❓ [Question] Why does the speed (fps) of torch-tensorrt perform so badly in `torch.multiprocessing`?

❓ Question

Hello, dear developer:
Thank your for your amazing job!
Why does the speed (fps) of torch-tensorrt perform so badly in torch.multiprocessing?
Currently I use torch.multiprocessing to create and run 3 Process (in 1 GPU) of resnet18, resnet50 and resnet101 at the same time. But I find their speeds of inference are worse than single process.
Here is my single process code:

# single process
import time

import torch
import tensorrt
import torch_tensorrt
from torchvision.models import resnet18, resnet50, resnet101
if __name__ == '__main__':
    # --------------------------------ResNet18---------------------------------------
    model0 = torch.jit.load("res18_trt_fp16.ts")
    inputs = [torch.randn((10, 3, 224, 224)).half().cuda()]
    print("Warm up ...")
    with torch.no_grad():
        for _ in range(10):
            features = model0(*inputs)
    torch.cuda.synchronize()
    t0 = time.time()
    with torch.no_grad():
        _ = model0(*inputs)
    torch.cuda.synchronize()
    t1 = time.time()
    print('res18: ', (t1 - t0) * 1000, 'ms')

    # --------------------------------ResNet50---------------------------------------
    model1 = torch.jit.load("res50_trt_fp16.ts")
    inputs = [torch.randn((10, 3, 224, 224)).half().cuda()]
    print("Warm up ...")
    with torch.no_grad():
        for _ in range(10):
            features = model1(*inputs)
    torch.cuda.synchronize()
    t0 = time.time()
    with torch.no_grad():
        _ = model1(*inputs)
    torch.cuda.synchronize()
    t1 = time.time()
    print('res50: ', (t1 - t0) * 1000, 'ms')

    # --------------------------------ResNet101--------------------------------------
    model2 = torch.jit.load("res101_trt_fp16.ts")
    inputs = [torch.randn((10, 3, 224, 224)).half().cuda()]

    with torch.no_grad():
        for _ in range(10):
            features = model2(*inputs)

    torch.cuda.synchronize()
    t0 = time.time()
    with torch.no_grad():
        res = model2(*inputs)
    torch.cuda.synchronize()
    t1 = time.time()
    print('res101: ', (t1 - t0) * 1000, 'ms')

The results are:

res18: 1.2104511260986328 ms
res50: 2.7513504028320312 ms
res101: 5.034923553466797 ms

And here is my multiprocessing code

# multiprocess
import pycuda.driver as cuda
import pycuda.autoinit

import os
import time
import numpy as np

import torch
import torch.multiprocessing as mp
import torch_tensorrt

def Worker1():
    print('Worker1 PID:', os.getpid())
    net = torch.jit.load("res18_trt_fp16.ts")
    x = torch.randn(10, 3, 224, 224).half().cuda()

    for i in range(10):
        _ = net(x)

    with torch.no_grad():
        while True:
            # infer
            torch.cuda.synchronize()
            t0 = time.time()

            results = net(x)

            torch.cuda.synchronize()
            t1 = time.time()
            print('Res18', (t1 - t0) * 1000, 'ms')

def Worker2():
    print('Worker2 PID:', os.getpid())
    net = torch.jit.load("res50_trt_fp16.ts")
    x = torch.randn(10, 3, 224, 224).half().cuda()

    for i in range(10):
        _ = net(x)

    with torch.no_grad():
        while True:
            # infer
            torch.cuda.synchronize()
            t0 = time.time()

            results = net(x)

            torch.cuda.synchronize()
            t1 = time.time()
            print('Res50', (t1 - t0) * 1000, 'ms')


def Worker3():
    print('Worker3 PID:', os.getpid())
    net = torch.jit.load("res101_trt_fp16.ts")
    x = torch.randn(10, 3, 224, 224).half().cuda()

    for i in range(10):
        _ = net(x)

    with torch.no_grad():
        while True:
            # infer
            torch.cuda.synchronize()
            t0 = time.time()

            results = net(x)

            torch.cuda.synchronize()
            t1 = time.time()
            print('Res101', (t1 - t0) * 1000, 'ms')



if __name__ == '__main__':
    mp.set_start_method('spawn', force=True)

    # create
    processes = [
        mp.Process(target=Worker1, args=()),
        mp.Process(target=Worker2, args=()),
        mp.Process(target=Worker3, args=()),
    ]

    # start
    for p in processes:
        p.start()

    # main loop
    while True:
        continue

BUT the results are (average):

Res18: 5.539894104003906 ms
Res50: 7.973670959472656 ms
Res101:13.53001594543457 ms

The results of multiprocessing are so wired. They are much slower than single process, which confuses me a lot.
Is there any way to fix them up or speed them up?
Thank you in advance!

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

• PyTorch Version (e.g., 1.0): 2.3.0 stable
• PyTorch-Tensorrt Version (e.g., 1.0): 2.3.0
• Tensorrt Version (e.g., 1.0): 10.0.1
• CPU Architecture: x64
• OS (e.g., Linux): ubuntu 22.04
• How you installed PyTorch (conda, pip, libtorch, source): pip install
• Build command you used (if compiling from source): No
• Are you using local sources or building from archives: No
• Python version: 3.11
• CUDA version: 12.1
• GPU models and configuration: RTX 4060
• Any other relevant information: No

No repository files or tests are named; start with the single-process and multiprocessing scripts included in the issue and reproduce their timing comparison. Check whether the slowdown is expected for concurrent inference on one GPU or indicates a Torch-TensorRT integration problem. Done means identifying a supported configuration or documenting the required code or runtime change.