NVIDIA / NVIDIA/DALI

Inplace operator support

Open
#4,533 2 comments 0 reactions 1 assignee View on GitHub

@mzient is already working on this.

Since Dec 22, 2022.

perf question
Dominant language
C++
Stars
5.8k
Forks
678
Avg merge
3d 1h
Merged PRs (30d)
27

Description

Hello, I wanted to ask whether it is possible to create in place operations. I have a pretty big DALI pipeline (in terms of image size) and I have to preprocess data, but each operation creates a copy of the data, that results in a DALI preprocessing pipeline with around 8Gb of memory consumption.

DALI version: 1.22.0dev

My neural network has an input size of 3 images with batchx3x5000x10000.

The pipeline consists of these steps:

  1. 3 Encoded 16-bit TIFF images (900Mb)
  2. nvidia.dali.fn.experimental.decoders.image (900Mb)
  3. nvidia.dali.fn.transpose (900Mb)
  4. nvidia.dali.fn.cast (1'800Mb)
  5. division operator (1'800Mb)
  6. nvidia.dali.fn.stack (1'800Mb)

Which takes around 8.1Gb of GPU memory just for pre-processing.

I am using DALI with Triton Inference Server and this is an issue because the TensorRT model is only around 1Gb memory and the pre-processing is 8x bigger. If some of the operations would be inplace it would greatly imporve the memory usage server-side. Is there a plan or a way to enable this?

Thanks in advance

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.