pytorch / pytorch/vision

Refactor the `forward` method of the RoiHeads into separate methods for easier customization

Open
#3,753 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement module: ops triage review
Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

🚀 Feature

Break up the code in the forward method into several sub-methods and allow custom loss and inference functions.

Motivation

I have a use case that requires implementing a different loss function for masks during training a Mask-RCNN, but designing this is somewhat messy right now. Ideally, I'd like to be able to inherit RoiHeads and then make a small change to a method like process_masks rather than overriding the whole forward method. (Or even simpler, just pass in custom loss functions during initialization).

Pitch

The structure of forward lends itself to re-factoring really simply.
forward could call, in sequence:
process_boxes(...)
process_masks(...)
process_keypoints(...)

This way, changing the behaviour requires only re-implementing one of these methods in the child class rather than all of forward. Optionally, these functions could use customized loss and inference functions, maybe passed into RoiHeads during initialization.

Alternatives

For my use case, it would also be sufficient to be able to optionally pass RoiHeads custom loss functions during initialization, but I think breaking it up is still a good idea because it makes other changes that do require changing how some of these processes are handled a little simpler and cleaner.

Another option would be to add the functions related to loss and inference to the class itself, so they could be directly overriden by a child class, but I think this may muddy the responsibilities of RoiHeads a little.

Additional context

Direct link to module: https://github.com/pytorch/vision/blob/master/torchvision/models/detection/roi_heads.py
I'm happy to take this on if approved.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with torchvision/models/detection/roi_heads.py and read the RoiHeads forward method and surrounding box, mask, and keypoint processing. Determine how the proposed sub-methods and customizable loss or inference functions fit the existing API, then verify that the refactoring preserves current training and inference behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.