Project-MONAI / Project-MONAI/MONAI

`apply_affine_to_boxes` does not handle flipping properly, due to the left-closed and right-open nature of the boxes

Open
#7,740 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

Describe the bug
apply_affine_to_boxes does not handle flipping properly, due to the left-closed and right-open nature of the boxes. That is to say, when a box is flipped, the left-closed and right-open coordinates become left-open and right closed, which should be maintained.

To be clarified, I understand that there's a flip_boxes function available, but I want to use this example for simplicity. Same issue exists for more complicated affine matrix, as long as containing flipping.

To Reproduce

import torch

from monai.apps.detection.transforms.box_ops import apply_affine_to_boxes
from monai.data import MetaTensor
import monai.transforms as mt

def main():
    for flip_axis in range(3):
        flip = mt.Flip(spatial_axis=flip_axis)
        x = torch.zeros(1, 1, 1, 1)
        flipped: MetaTensor = flip(x)
        box = torch.tensor([[0, 0, 0, 1, 1, 1]])
        box_flipped = apply_affine_to_boxes(box, flipped.affine.inverse())
        print(box_flipped)

if __name__ == '__main__':
    main()

Expected behavior
Produce the same results as flip_boxes.

Actual Results

tensor([[-1,  0,  0,  0,  1,  1]])
tensor([[ 0, -1,  0,  1,  0,  1]])
tensor([[ 0,  0, -1,  1,  1,  0]])

Suggestion
My suggestion for the fix is to convert the box coordinates to be closed on the both sides before applying the affine matrix, and convert them back to the desired format after applying the affine.

Additional Context

If I understand correctly, the left-closed and right open nature is suggested here:
https://github.com/Project-MONAI/MONAI/blob/e1a69b03c86ce065db2816b696ea4a6b57d46435/monai/data/box_utils.py#L40-L45

Feature Request

This is not about the major topic of this issue, but by the way, I would like to mention that in the code above I calculate the inverse of the affine matrix. Currently, I only find this way works to apply arbitrary affine transform to boxes. It would be nice if apply_affine_to_boxes could do it for be internally by solving a linear equation for numerical stability.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with apply_affine_to_boxes in monai/apps/detection/transforms/box_ops.py and compare its behavior with flip_boxes. Review the left-closed and right-open convention described in monai/data/box_utils.py, then use the reproduction case to verify that flipped boxes match flip_boxes for each axis.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.