pytorch / pytorch/vision

Video decoding tests are never run on CI

Open
#8,162 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

The testing of the video decoders is in a catastrophic state. Almost all tests for read_video, VideoReader or the GPU decoder are being skipped on our CI, for one reason or another.

  • The GPU tests are never run because we never build with support for the nvidia lib. (I guess it's sort of OK considering the state of that GPU decoder itself... we should probably just remove that decoder entirely.)

  • The other tests (read_video and VideoReader) are never run either. @pmeier's https://github.com/pytorch/vision/pull/8157 suggests that the video ops cannot be loaded with the following error:

Failed to load video Python extension: '/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /opt/conda/envs/ci/lib/./libopenh264.so.5)'

Although for some Python versions, we don't even get an error message.

  • The tests in test_io.py are being run. They only test read_video on the pyav backend AFAICT.

The lack of test for those decoders isn't new and has been going on for a LONG time. In https://github.com/pytorch/vision/pull/7646 we already noticed part of the problem.

Part of the reason this hasn't been flagged earlier is because our tests are setup incorrectly: they don't tell you if they can't be run! If for some reason the ops cannot be loaded, or if pyav isn't present, or if there was a problem with the ffmpeg build, then the tests are simply skipped. I understand this is because we can't be testing those APIs on all platforms, but one bad consequence is that when the build fails (and it FAILS), then the tests are just skipped and CI is still happy with that, so we never know. We think the tests are green when in reality they were never run.

So, beyond bringing those tests back, we have to find a way to be properly alerted when those tests aren't run when we expect them to be run.

On https://github.com/pytorch/vision/pull/8161 we have decided to at least stop pretendying, and explicitly excluded those files from being tested until the situation gets resolved. We should probably start with bringing back the read_video tests.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the read_video tests in test_io.py and review the exclusions described in PR 8161, along with the VideoReader and GPU decoder test paths. Check why these tests are skipped or cannot load in CI. Done means the intended read_video tests run in CI and unexpected decoder or dependency failures are reported rather than silently skipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, computer-vision, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.