Inplace operator support
@mzient is already working on this.
Since Dec 22, 2022.
- Dominant language
- C++
- Stars
- 5.8k
- Forks
- 678
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
Hello, I wanted to ask whether it is possible to create in place operations. I have a pretty big DALI pipeline (in terms of image size) and I have to preprocess data, but each operation creates a copy of the data, that results in a DALI preprocessing pipeline with around 8Gb of memory consumption.
DALI version: 1.22.0dev
My neural network has an input size of 3 images with batchx3x5000x10000.
The pipeline consists of these steps:
- 3 Encoded 16-bit TIFF images (900Mb)
- nvidia.dali.fn.experimental.decoders.image (900Mb)
- nvidia.dali.fn.transpose (900Mb)
- nvidia.dali.fn.cast (1'800Mb)
- division operator (1'800Mb)
- nvidia.dali.fn.stack (1'800Mb)
Which takes around 8.1Gb of GPU memory just for pre-processing.
I am using DALI with Triton Inference Server and this is an issue because the TensorRT model is only around 1Gb memory and the pre-processing is 8x bigger. If some of the operations would be inplace it would greatly imporve the memory usage server-side. Is there a plan or a way to enable this?
Thanks in advance
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.