pytorch / pytorch/vision

Current limitation on transforms

Open
#3,224 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement module: transforms
Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

The torchvision transforms now have 2 backends (PIL and Tensor), here are some functional mismatch between them and some may-be-useful features that neither of them support. Details are listed in transforms.py and functional.py.

Supported by PIL but not Tensor:

  1. Fill value for pad and random crop.
  2. Tensor images do not support many modes due to lack of metadata (maybe not possible to address). For instance, the adjust_* functions and autoaugment related functions.
  3. Tensors only support 3 interpolation modes (bilinear, linear, nearest).
  4. Tensors only support transformations on RGB images

Crop with crop size larger than the original image. #3297 Solved by #3333

Supported by Tensor but not PIL:

  1. Normalize (probably of no use for PIL images).
  2. Erase.

Supported by neither:

  1. adjust_gamma() and adjust_hue() do not support images with transparency.
  2. Subpixel translations. #3293

Not supported by torchscript (mostly not possible given the current jit support):

  1. single value inputs in Pad(fill), RandomCrop(padding), Resize(size), RandomResizedCrop(size).
  2. PIL and Tensor conversions.
  3. Compose, RandomOrder, RandomChoice.
  4. Lambda.

It is just a draft, let me know if I forget anything. cc @vfdev-5 @datumbox

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing torchvision/transforms/transforms.py and torchvision/transforms/functional.py, then compare the listed PIL, Tensor, and TorchScript limitations. The issue is a broad inventory rather than a scoped task, so a contributor would need to select one limitation and define its expected behavior and tests before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.