NVIDIA / NVIDIA/TensorRT

model inputs and out put problem

Open
#3,814 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

My model has two inputs, one size is (1, 5,256,256), the other size is (1,3,512,512), there are eight outputs, I need to use the trt model to achieve reasoning, and get reasoning results. The relevant code I have looked up so far is for one input, and I want to know how to write code that has multiple inputs and outputs for the model.

Mycode

runtime= trt.Runtime(trt.Logger())
inference_time = []

with open('/home/AGX-exte/qry/AGX_EXP/model_oxnn/model.trt','rb') as f,runtime.deserialize_cuda_engine(f.read()) as engine:

context = engine.cteate_excuteion_context()

for i,data in enumerate(test_data):
    
    context.set_binding_shape(0, (1, 5, 256, 256))
    
    context.set_binding_shape(1, (1, 3, 512, 512))

    grd, sat, gt, gt_ori, gt_orientation, angle = (iterm for iterm in data)

    input_data1 = grd

    input_data2 = sat

    #host inpiut
    input_data1_np = input_data1.numpy()
    input_data2_np = input_data2.numpy()
    #host output
    '''
    output size is:
        (1, 262144)
        (1, 1, 512, 512)
        (1, 2, 512, 512)
        (1, 12, 16, 16)
        (1, 12, 32, 32)
        (1, 12, 64, 64)
        (1, 12, 128, 128)
        (1, 12, 256, 256)
    '''
    output1 = np.empty((1, 262144),dtype = np.float32)
    output2 = np.empty((1, 1, 512, 512),dtype = np.float32)
    output3 = np.empty((1, 2, 512, 512),dtype = np.float32)
    output4 = np.empty((1, 12, 16, 16),dtype = np.float32)
    output5 = np.empty((1, 12, 32, 32),dtype = np.float32)
    output6 = np.empty((1, 12, 64, 64),dtype = np.float32)
    output7 = np.empty((1, 12, 128, 128),dtype = np.float32)
    output8 = np.empty((1, 12, 256, 256),dtype = np.float32)
    #alloc input Mem
    input_device_buffer = [cuda.men_alloc(input_data1_np.nbytes),cuda.men_alloc(input_data2_np.nbytes)]

    cuda.memcpy_htod(input_device_buffer[0], input_data1_np)
    cuda.memcpy_htod(input_device_buffer[1], input_data2_np)
    #alloc Mem
    output_device_buffer = [cuda.mem_alloc(output1.nbytes), cuda.mem_alloc(output2.nbytes),cuda.mem_alloc(output3.nbytes),cuda.mem_alloc(output4.nbytes),
                    cuda.mem_alloc(output5.nbytes),cuda.mem_alloc(output6.nbytes),cuda.mem_alloc(output7.nbytes),cuda.mem_alloc(output8.nbytes)
                    ]

    # do inference
    begin = time.time()
    context.execute_v2(bindings=[int(i) for i in input_device_buffer] + [int(j) for j in output_device_buffer])
    end = time.timinference_timee()
    inference_time.append(begin-end)
    cuda.memcpy_dtoh(output1, output_device_buffer[0])
    cuda.memcpy_dtoh(output2, output_device_buffer[1])
    cuda.memcpy_dtoh(output3, output_device_buffer[2])
    cuda.memcpy_dtoh(output4, output_device_buffer[3])
    cuda.memcpy_dtoh(output5, output_device_buffer[4])
    cuda.memcpy_dtoh(output6, output_device_buffer[5])
    cuda.memcpy_dtoh(output7, output_device_buffer[6])
    cuda.memcpy_dtoh(output8, output_device_buffer[7])

inference_time = np.array(inference_time)
np.save('',inference_time)`

Operating System:
linux
Python Version (if applicable):
3.6.9

Relevant Files

Model link:

Steps To Reproduce

Commands or scripts:

Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TensorRT Python code around engine.deserialize_cuda_engine, set_binding_shape, and execute_v2, then verify the engine's binding order and declared input and output shapes against the allocated buffers. Done means the provided model runs with both inputs and all eight outputs are retrieved successfully, but the issue provides no model link, traceback, or reproducible commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.