facebookresearch / facebookresearch/detectron2

Custom transform for augmenting image using mask in detectron2

Open
#5,018 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

In the process of data augmentation, I am trying to modify the image based on the corresponding mask. I need to get the background from the image, which requires knowing the foreground (mask) in advance.

Therefore, I am looking for a `Transform` that can provide image and mask as input to my function. For example, previously, I used [ColorTransform](https://detectron2.readthedocs.io/en/latest/modules/data_transforms.html#detectron2.data.transforms.ColorTransform), which takes a callable and provides an image as input to this callable. See an example below:

```
def apply_blur(img):
blur_img = cv2.GaussianBlur(img, (3, 3), 0)
return blur_img

class MyTrainer(DefaultTrainer):
@classmethod
def build_train_loader(cls, cfg):
augmentations = [
T.RandomFlip(),
T.ColorTransform(apply_blur),
]
mapper = DatasetMapper(cfg, is_train=True, augmentations=augmentations)
return build_detection_train_loader(cfg, mapper=mapper)
```

I found [AugInput](https://detectron2.readthedocs.io/en/latest/modules/data_transforms.html#detectron2.data.transforms.AugInput), but it expects image, mask, etc. as input. Instead, is there something like the following exists?

```
def my_transform(img, mask):
# do something on image using mask and return image
return img

...
def build_train_loader(cls, cfg):
augmentations = [
T.RandomFlip(),
T.CustomTransform(my_transform),
]
...
```

Thank you very much

----------------------------

## Instructions To Reproduce the 🐛 Bug:
1. Full runnable code or full changes you made:
```
If making changes to the project itself, please use output of the following command:
git rev-parse HEAD; git diff

```
2. What exact command you run:
3. __Full logs__ or other relevant observations:
```

```
4. please simplify the steps as much as possible so they do not require additional resources to
run, such as a private dataset.

## Expected behavior:

If there are no obvious error in "full logs" provided above,
please tell us the expected behavior.

## Environment:

Provide your environment information using the following command:
```
wget -nc -q https://github.com/facebookresearch/detectron2/raw/main/detectron2/utils/collect_env.py && python collect_env.py
```

If your issue looks like an installation issue / environment issue,
please first try to solve it yourself with the instructions in
https://detectron2.readthedocs.io/tutorials/install.html#common-installation-issues

Contributor guide

Open the contributing guide

Research direction

Start by reading the documented ColorTransform and AugInput APIs, then trace how DatasetMapper applies augmentations from build_train_loader. Compare the requested image-and-mask callable with the existing transform entry points; done would require a decided, supported approach for mask-aware augmentation and corresponding documentation or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
opencv, python
Domain
computer-vision
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.