memory use continuously increasing

I'm working from the tutorials for integrating DALI with pytorch, aiming to train models on ImageNet. But I think I'm running into the "memory leak" / "continuously growing memory" issues mentioned in (#344, and #278), although none of the suggestions in those issues solved my problem.

I'm using Nvidia Dali 0.6.1, with ubuntu 16.04, cuda10.0, cudnn7.4.1, pytorch v1.0.0

I'm using a hybrid pipline and the DALIGenericIterator from the pytorch plugin.

from nvidia.dali.pipeline import Pipeline
from nvidia.dali.plugin.pytorch import DALIGenericIterator, DALIClassificationIterator
import nvidia.dali.ops as ops
import nvidia.dali.types as types

def ram_use():
    import psutil
    pid = os.getpid()
    py = psutil.Process(pid)
    memoryUse = py.memory_info()[0] / 2. ** 30  # memory use in GB...I think
    return memoryUse

class ImageNetPipeline(Pipeline):
    def __init__(self, image_dir, batch_size, num_threads, device_id, exec_async=True):
        super(ImageNetPipeline, self).__init__(batch_size, num_threads, device_id, seed = 12, exec_async=exec_async)
        self.input = ops.FileReader(file_root = image_dir, random_shuffle = True, initial_fill = 21)
        self.decode = ops.nvJPEGDecoder(device = "mixed", output_type = types.RGB)
        self.resize = ops.Resize(device = "gpu", resize_shorter=224)
        self.centerCrop = ops.Crop(device = "gpu", crop=(224,224))
        self.norm = ops.NormalizePermute(device = "gpu",
                                            height = 224,
                                            width = 224,
                                            mean = [x*255 for x in [0.485, 0.456, 0.406]],
                                            std = [x*255 for x in [0.229, 0.224, 0.225]])
    
    def define_graph(self):
        jpegs, labels = self.input()
        images = self.decode(jpegs)
        images = self.resize(images)
        images = self.centerCrop(images)
        images = self.norm(images)
        
        # images are on the GPU
        return (images, labels)

N = 2 # number of GPUs
BATCH_SIZE = 128  # 128, batch size per GPU
ITERATIONS = 32
NUM_THREADS = 8

train_dir = "/data/local_hdd/ImageSets/imagenet/ILSRC2012/train"

pipes = [ImageNetPipeline(image_dir=train_dir, batch_size=BATCH_SIZE, num_threads=NUM_THREADS, device_id=device_id) for device_id in range(N)]
pipes[0].build()
train_iter = DALIGenericIterator(pipes, ['data', 'label'], pipes[0].epoch_size().popitem()[1])

When I iterate through the dataset (not model training, just iterating), things go blazingly fast (5000 images/s), but only up until about 90% of the dataset has been loaded (so close!), at which things slow to a near standstill. During that time, my RAM useage steadily increases by 6-7 GB, (e.g., starting from 5GB to about 12.5GB). I'm not sure why things stall at 12.5GB (the machine has 128 GB of RAM), but this is consistent across many attempted runs.

batch_no = 0
count = 0
for data in train_iter:
    for batch in data:
        batch_no += 1
        count += batch['data'].shape[0]
    if batch_no % 100 == 0:
        print(batch_no, count, ram_use())

I made my own copy of DALIGenericIterator to determine the source of the issues. It seems that calling p._share_outputs() increases the memory use. If I "simuluate" iterations without this function call (by calling p._share_outputs once during the first batch, storing the outputs, and just working with the same outputs on each iteration), then the memory doesn't grow.

Is it expected that memory use would grow on each iteration/call to p._share_outputs()?

Is it possible that p._release_outputs() is not releasing memory?

Since _share_outputs, _release_outputs are core functions, I wasn't sure how to further debug this issue.

Many thanks in advance for your help.

Reproduce the ImageNet iteration with the shown ImageNetPipeline and DALIGenericIterator setup, then trace the plugin entry points _share_outputs() and _release_outputs(). Compare memory usage across iterations and confirm that a completed run no longer stalls near the end of the dataset or shows unbounded growth.