pytorch / pytorch/vision

Add batch visualization function to `torchvision.utils`

Open
#8,201 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
17.9k
Forks
7.3k
Avg merge
1d 15h
Merged PRs (30d)
13

Description

🚀 The feature

Currently vision models commonly return dictionary

model = fasterrcnn_resnet50_fpn_v2(
    weights=FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1,
)
dog_int = io.read_image("dog.jpeg")
batch = f.convert_image_dtype(dog_int)
response = model(batch.unsqueeze(0))
# [{'boxes': tensor([[ 61.9160,  49.2223, 185.7204, 184.4109],
#          [143.5477, 143.5992, 175.1468, 184.4712],
#          [136.6003, 217.3529, 161.4755, 223.8254]]),
#  'labels': tensor([18, 20, 57]),
#  'scores': tensor([0.9989, 0.1069, 0.0699])}]

This is visualized by torchvision by

dog_image = utils.draw_bounding_boxes(
    dog_int, response[0]["boxes"][response[0]["scores"] > 0.3]
)
f.to_pil_image(dog_image)
Motivation, pitch

However, there is no currenly a way to visualize whole batch with a function in utilities, eg.
continuing from previous

with torch.no_grad():
    imgs = [img.detach().clone() for _ in range(16)]
    model.eval()
    response = model(imgs)
# The input is float tensor batch
# output is equal length list of output dictionaries

There is no currenly available method to visualize this batch for user.

I suggest an utils function

def visualize_batch(image_batch: Tensor | list[Tensor], batch_response: list[dict[str, Tensor]], **visualization_arguments):
    """Function visualizes image batch in a suitable grid and returns result as a tensor

       Arguments:
            image_batch (Tensor): Float tensor batch, internally transform to uint8 for visualization utils
            batch_response: List of response dictionaries, works for both mask and rcnn models as well as training batches by inferring correct behavior from dictionary keys.
    """
    # ... implementation

where keyword arguments relate to arguments in current utils functions.

Alternatives

User could follow utilities example at example which implements matplotlib function show. This is non-ideal, as users seems to have need for similar functionality without related boilderplate code.

Additional context

I am willing to contribute given green light.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing torchvision.utils visualization functions and the linked visualization example. Compare their arguments with the proposed visualize_batch inputs, then clarify supported response dictionaries, grid behavior, and how completion will be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.