GPU memory increases during trainning

I am using tfrecord to load data, the GPU memory increases from about 11G to 15G during trainning in one epoch. So I have to decrease the batch_size in case OOM error occurs.
However, when default pytorch dataloader is used, the GPU memory does not increase dramatically.

The codes that used to get input and lables are shown below

trainingPipes = TFRecordPipeline(tfrecord_path=train_tfrecord_path,
                                     tfrecord_idx_path=train_tfrecord_idx,
                                     batch_size=batch_size, 
                                     num_threads=4, 
                                     device_id = 0,
                                     crop_size = crop_size,
                                     image_size = img_size,
                                     randomFlip = True,
                                     random_shuffle=True)

        trainingPipes.build()

        trainloader = DALIClassificationIterator(
                      trainingPipes,
                      size=int(trainingPipes.epoch_size("Reader"))
        ) 

        for i, data in enumerate(trainloader):

            images = data[0]["data"].cuda()

            labels_ = data[0]["label"].cpu().numpy()

            labels = []
            for label in labels_:
                labels.append(label[0])

            labels = torch.FloatTensor(labels).long().cuda()

numpy array is used to rearrange the lables because data[0]["label"] is a list of list. It is in the format [ [label_1], [label_2], [label_3]...]

Start by reproducing the reported memory growth around TFRecordPipeline and DALIClassificationIterator, using the shown training loop and comparing it with the default PyTorch dataloader. Track GPU memory from the start through one epoch and isolate whether the increase comes from loading, label conversion, or iteration; done means the cause is identified and the reported growth is resolved or clearly documented.