torchvision.io.read_video mirrors videos of BDD100k and sometimes rotates them 180 degrees

🐛 Bug

The function torchvision.io.read_video(path, start_pt, end_pt, pts_unit='sec') flips videos horizontally and for around half of the videos, also rotates them 180 degrees. The videos in question are from the dataset BDD100k. I believe they have been collected with an iPhone, and are in .mov format.

To Reproduce

Steps to reproduce the behavior:

import torch
import torchvision as tv
import matplotlib.pyplot as plt

vid1, _, _ = tv.io.read_video('data/bdd100k/bdd100k/videos/train/011f8481-b14725fa.mov', 9.5, 10.5, pts_unit='sec')
vid2, _, _ = tv.io.read_video('data/bdd100k/bdd100k/videos/train/019c8305-3a3be0d6.mov', 9.5, 10.5, pts_unit='sec')

# vid1.size() -> torch.Size([31, 1280, 720, 3])

plt.imshow(vid1[0].permute(1, 0, 2))
plt.show() # See first figure
plt.imshow(vid2[0].permute(1, 0, 2))
plt.show() # See second figure

Figures:

Videos when opened with QuickTime:

ffprobe provides:

> ffprobe data/bdd100k/bdd100k/videos/train/011f8481-b14725fa.mov
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.27)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.3.1_4 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'data/bdd100k/bdd100k/videos/train/011f8481-b14725fa.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 512
    compatible_brands: qt
    encoder         : Lavf57.71.100
  Duration: 00:00:40.09, start: 0.000000, bitrate: 3949 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, smpte170m/bt709/bt709), 720x1280, 3946 kb/s, 30.06 fps, 60 tbr, 19200 tbn, 38400 tbc (default)
    Metadata:
      rotate          : 270
      handler_name    : VideoHandler
      encoder         : H.264
    Side data:
      displaymatrix: rotation of 90.00 degrees

> ffprobe data/bdd100k/bdd100k/videos/train/019c8305-3a3be0d6.mov
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.27)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.3.1_4 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'data/bdd100k/bdd100k/videos/train/019c8305-3a3be0d6.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 512
    compatible_brands: qt
    encoder         : Lavf57.71.100
  Duration: 00:00:40.16, start: 0.000000, bitrate: 4085 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, smpte170m/bt709/bt709), 720x1280, 4082 kb/s, 30.03 fps, 29.97 tbr, 19200 tbn, 38400 tbc (default)
    Metadata:
      rotate          : 90
      handler_name    : VideoHandler
      encoder         : H.264
    Side data:
      displaymatrix: rotation of -90.00 degrees

Expected behavior

I expect tv.io.read_video to provide images similar to what would be displayed in QuickTime.

Environment

• PyTorch / torchvision Version (e.g., 1.0 / 0.4.0): 1.7.1 / 0.8.2
• OS (e.g., Linux): macOS Big Sur
• How you installed PyTorch / torchvision (conda, pip, source): Conda
• Build command you used (if compiling from source): N/A
• Python version: 3.8.5
• CUDA/cuDNN version: N/A
• GPU models and configuration: N/A
• Any other relevant information:

Best regards,
Joakim

cc @bjuncek

Start at the torchvision.io.read_video entry point and reproduce the issue with the two BDD100k .mov files named in the report. Compare the decoded frames with QuickTime and the rotation metadata reported by ffprobe. Done means videos with these metadata orientations are returned in the expected visual orientation without unwanted mirroring or rotation.