Building (video) pipeline slow with high number of samples

Hello. I noticed that the time required for building the pipeline grows linearly with the number of sample in the video list. I am using more or less this code:

train_loader = DALIGenericIterator(
    pipelines=[
        MyPipeline(
            sample_list_path=video_file_list,  # this is the txt file where each line is 'path label start end'
            shuffle=True,
            batch_size=self.batch_size,
            num_threads=2,
        )
    ],
    ...
)

I noticed that executing the above snippet can take long when number of videos is big. I am referring only to the building time. For example:
1000 samples (rows) in the txt file -> 12.81 sec
2000 samples (rows) in the txt file -> 24.52 sec
My dataset is way bigger than that (100x) so this linearly increasing setup time is not a viable solution.

Are these numbers expected in your experience? Maybe I am doing something wrong while configuring the operators... I am using DALI 1.20.0 from official Nvidia 22.12 container.