Errors when reading .webm and converted mp4 files

Hi,

I'm trying to using DALI to train on the something-something-v2 dataset. The video files are all .webm files encoded using the VP9 codec. I'm not sure if the DALI VideoReader supports .webm files, but when I try to use it with skip_vfr_check=False (since otherwise it errors out saying VFR is not supported but the video seems to be CFR with frame rate 12/1), it ends up eating up a lot of RAM (e.g. uses ~100GB of RAM for 1000 ~70-90KB videos during pipeline.build()). After awhile, I get the following error:

>>> singularity exec --nv docker://nvcr.io/nvidia/pytorch:21.02-py3 python example.py
Traceback (most recent call last):
  File "example.py", line 37, in <module>
    pipeline.build()
  File "/opt/conda/lib/python3.8/site-packages/nvidia/dali/pipeline.py", line 481, in build
    self._pipe.Build(self._names_and_devices)
RuntimeError: Critical error when building pipeline:
Error when constructing operator: VideoReader encountered:
std::bad_alloc
Current pipeline object is no longer valid.

I tried to convert the .webm videos to a fixed frame rate .mp4 format using this script:

import os
import os.path as osp
import argparse
import glob
import multiprocessing as mp
from tqdm import tqdm

def worker(args):
    filename, output_dir = args
    f = osp.basename(filename)
    f = osp.splitext(f)[0] + '.mp4'
    output_filename = osp.join(output_dir, f)
    cmd = f'ffmpeg -i "{filename}" -vf "crop=trunc(iw/2)*2:trunc(ih/2)*2" -r 12/1 -c:v libx264 -c:a copy -crf 23 "{output_filename}" >/dev/null 2>&1'
    os.system(cmd)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-i', '--input_dir', type=str, default='20bn-something-something-v2')
    parser.add_argument('-o', '--output_dir', type=str, default='converted')
    args = parser.parse_args()

    os.makedirs(args.output_dir)

    files = glob.glob(osp.join(args.input_dir, '*.webm'))
    print(f"Found {len(files)} video files")
    files = [(f, args.output_dir) for f in files]

    with mp.Pool(mp.cpu_count()) as p:
        r = list(tqdm(p.imap(worker, files), total=len(files)))

And the video loader works for some batches but eventually errors out on failing to decode specific .mp4 files. Those files do seem fine, as I'm able to view them on my video players, and in the browser. Below is the error I get:

>>> singularity exec --nv docker://nvcr.io/nvidia/pytorch:21.02-py3 python example.py
0 video with shape torch.Size([32, 3, 16, 224, 224])
1 video with shape torch.Size([32, 3, 16, 224, 224])
2 video with shape torch.Size([32, 3, 16, 224, 224])
3 video with shape torch.Size([32, 3, 16, 224, 224])
4 video with shape torch.Size([32, 3, 16, 224, 224])
5 video with shape torch.Size([32, 3, 16, 224, 224])
6 video with shape torch.Size([32, 3, 16, 224, 224])
7 video with shape torch.Size([32, 3, 16, 224, 224])
8 video with shape torch.Size([32, 3, 16, 224, 224])
9 video with shape torch.Size([32, 3, 16, 224, 224])
10 video with shape torch.Size([32, 3, 16, 224, 224])
11 video with shape torch.Size([32, 3, 16, 224, 224])
12 video with shape torch.Size([32, 3, 16, 224, 224])
13 video with shape torch.Size([32, 3, 16, 224, 224])
14 video with shape torch.Size([32, 3, 16, 224, 224])
/opt/dali/dali/operators/reader/nvdecoder/nvdecoder.cc:157: Unable to decode file /home/wilson/data/datasets/something-something/converted/180116.mp4
terminate called after throwing an instance of 'dali::CUDAError'
  what():  CUDA driver API error CUDA_ERROR_UNKNOWN (999):
unknown error
Aborted (core dumped)

I ran the following example.py script in nvidia's latest pytorch container to produce the results above, in addition to downloading some videos from something-something-v2.

import os
import os.path as osp

from nvidia.dali.pipeline import Pipeline
import nvidia.dali.ops as ops
from nvidia.dali.plugin import pytorch

class VideoPipe(Pipeline):
    def __init__(self, batch_size, num_workers, device_id, seed):
        super().__init__(batch_size, num_workers, device_id, seed)

        root = '/home/wilson/data/datasets/something-something/converted'
        files = os.listdir(root)
        files = [osp.join(root, f) for f in files]

        self.input = ops.VideoReader(device='gpu', filenames=files,
                                     sequence_length=16, normalized=False,
                                     shard_id=0, num_shards=1, random_shuffle=True,
                                     initial_fill=batch_size, skip_vfr_check=True)
        self.crop = ops.Crop(device='gpu', crop=(224, 224))
        self.tp = ops.Transpose(device='gpu', perm=[3, 0, 1, 2])

    def define_graph(self):
        output = self.input(name='Reader')
        output = self.crop(output)
        output = self.tp(output)
        return output


batch_size = 32
num_workers = 4
device_id = 0
seed = 0

pipeline = VideoPipe(batch_size, num_workers, device_id, seed)
pipeline.build()

loader = pytorch.DALIGenericIterator(pipeline, ['video'], reader_name='Reader',
                                     last_batch_policy=pytorch.LastBatchPolicy.DROP,
                                     auto_reset=True)

for i, batch in enumerate(loader):
    print(i, 'video with shape', batch[0]['video'].shape)

I'm not too familiar with video decoding / encoding, so maybe I just used ffmpeg incorrectly, but any help on this would be appreciated!

Reproduce the failures with example.py and the listed VideoReader configuration, including the .webm files and converted/180116.mp4. Start at VideoReader and dali/operators/reader/nvdecoder/nvdecoder.cc, then compare the build-time bad_alloc with the later decode failure. Done means the reported inputs load without either error, or the failing input and decoder limitation are clearly identified.