pytorch / pytorch/vision

Restructuring torchvision tests

Open
#6,020 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

Motivation

Our tests are getting a bit lengthy and considering we are adding new features, we are actually placing multiple tests in single file.
This leads to problems as the tests become harder to refactor and harder to read, manage. Standalone files for tests or grouped according to purpose are easier to anaylse and debug as well.

Our longest test files

Copy paste-able command

find test -name "*.py" | xargs -I {} wc -l {} | sort -nr | head -n 10
oke@psycon:~/Aditya/PyTorch/vision$ find test -name "*.py" | xargs -I {} wc -l {} | sort -nr | head -n 10
2675 test/test_datasets.py
2242 test/test_transforms.py
1535 test/builtin_dataset_mocks.py
1528 test/test_ops.py
1343 test/test_functional_tensor.py
1339 test/test_prototype_transforms_functional.py
1254 test/test_video_reader.py
982 test/test_transforms_tensor.py
954 test/datasets_utils.py
942 test/test_models.py

Many splits are possible that can make the tests smaller and easier to manage

E.g. 

test_models.py
-----  test_classification_models.py
----- test_detection_models.py
----- test_video_models.py

test_ops.py
------ test_post_processors.py
------ test_losses.py
------ test_layers.py

Possibilities are many, we can have multiple files with better grouping.

Pitch

We should think of grouping tests in logically similar way together. or consider creating test folder a package. See
https://github.com/pytorch/vision/pull/4436 . Earlier we have faced problems doing the same.

From discussion with Phillip offline
Considering we don't Pin pytest. We can make use of https://github.com/pytest-dev/pytest/pull/9134

With the python path param, we can probably make use of common_utils.py file as well. So that we can test files without hurting other scenarios which were mentioned in the comments.

Alternatives

This is quite tricky but very helpful as we don't want tests to hurt us later.

Reference

https://github.com/pytorch/vision/pull/4436#issuecomment-923131237

https://github.com/pytorch/vision/pull/4436#issuecomment-924005657

https://github.com/pytorch/vision/pull/4436#issuecomment-924250093

cc @NicolasHug @datumbox @pmeier

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided find test command and reviewing the longest files, including test/test_datasets.py, test/test_transforms.py, test/test_ops.py, and test/test_models.py. Read the referenced pull request discussions, especially PR 4436, before choosing a grouping. Done means tests are logically split into manageable files while preserving behavior and avoiding the earlier test-package problems.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.