NVIDIA / NVIDIA/TensorRT

[Question] Is `IOutputAllocator::reallocateOutput` guaranteed to be called before `context->enqueueV3` returns?

Open
#3,875 5 comments 0 reactions 1 assignee View on GitHub

@zerollzeng is already working on this.

Since May 26, 2024.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

I cannot find any information regarding when IOutputAllocator::reallocateOutput is called with respect to context->enqueueV3. Is there any guarantee this function is called before enqueueV3 returns or should I explicitly synchronize stream?

In other words, in the following pseudo-code:

context_->setOutputAllocator(name, allocator);
// ...
context->enqueueV3(stream);
cudaStreamSynchronize(stream); // <-- Is this necessary?

// Memcpy devie -> host; Is it valid to ask allocator for device buffer without stream synchronization?
cudaMemcpyAsync(hostBuffer, allocator->getDeviceBuffer(), ...);

Should I explicitly synchronize the stream after enqueueV3 for device allocator->getDeviceBuffer() to be valid? Or is allocator->reallocateOutput guaranteed to be called before enqueueV3 returns, in which case stream synchronization is unnecessary?

Environment

TensorRT Version:

NVIDIA GPU:

NVIDIA Driver Version:

CUDA Version:

CUDNN Version:

Operating System:

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Model link:

Steps To Reproduce

Commands or scripts:

Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.