pytorch / pytorch/vision

[feature request] Make `transforms.functional_tensor` functions differential w.r.t. their parameters

Open
#5,000 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement module: transforms
Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

🚀 The feature

Make operations in torchvision.transforms.functional_tensor differential w.r.t. hyper-parameters, which is helpful for Faster AutoAugment search (hyper-parameters are learnable parameters via backward). (while keeping the backward compatibility to previous codes)

Some operations are not differential (e.g., Posterize), which might require users to write their own implementations.

Motivation, pitch

The main motivation is for research purpose. Faster Autoaugment proposes to search for augment architectures using a DARTS-like framework, and all magnitudes and weights are trainable parameters. This requires all operations to have gradients w.r.t. magnitudes. This idea provides a faster search strategy as state-of-the-art AutoAugment policy search algorithms.
This work has been maintained by autoalbument and applied on some industrial scenarios from their document claims.

I think adding the backward feature wrt magnitudes would be more convenient and support future research as well.

Alternatives

No response

Additional context

Linked PR: #4995

cc @vfdev-5 @datumbox

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in torchvision.transforms.functional_tensor and review the operations whose hyper-parameters need gradients, using the Faster AutoAugment motivation and linked PR #4995 as context. Preserve backward compatibility with existing code; done means supported operations are differentiable with respect to their magnitudes while acknowledging operations such as Posterize may remain unsupported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.