pytorch / pytorch/vision

[fyi][video] Video clip creation fails on YFCC

Open
#1,327 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug help wanted module: io module: video
Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

Culprit:

known bug in H264 decoding

For certain filetypes - specifically some related to YFCC100m dataset - there is an error whilst creating the VideoClips object. Not 100% sure what is the best way of fixing the issue (other than try-catch in read_video_timestamps.

Even if the fix is not critical, we should make sure that is caught in C++ or future versions.


Sample YFCC video in question:

27b66931edb88a5f7e6c756ea65fd8.mp4
there are at least 50 of them, probably more.

Minimal repro example

video_list = ['~/yfcc100m/mp4/27b/669/27b66931edb88a5f7e6c756ea65fd8.mp4']
VideoClips(video_list, 8, 1)

or even simpler

import av
container = av.open('~/yfcc100m/mp4/27b/669/27b66931edb88a5f7e6c756ea65fd8.mp4')
for idx, frame in enumerate(container.decode({'video': 0})):
    pass

Output error stack:

Invalid NAL unit size (34173 > 22144).
Error splitting the input into NAL units.
stream 1, offset 0x16baf66: partial file
Invalid NAL unit size (24747 > 6180).
Error splitting the input into NAL units.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the VideoClips construction path and read_video_timestamps, then reproduce the failure using 27b66931edb88a5f7e6c756ea65fd8.mp4 and the av.open decoding example from the issue. Determine how the H264 decoding error should be handled, and verify that creating VideoClips no longer fails unexpectedly for the affected YFCC files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.