facebookresearch / facebookresearch/detectron2
Custom transform for augmenting image using mask in detectron2
- Dominant language
- Python
- Stars
- 34.7k
- Forks
- 7.9k
- PR merge metrics
- No merged PRs in 30d
Description
In the process of data augmentation, I am trying to modify the image based on the corresponding mask. I need to get the background from the image, which requires knowing the foreground (mask) in advance.
Therefore, I am looking for a `Transform` that can provide image and mask as input to my function. For example, previously, I used [ColorTransform](https://detectron2.readthedocs.io/en/latest/modules/data_transforms.html#detectron2.data.transforms.ColorTransform), which takes a callable and provides an image as input to this callable. See an example below:
```
def apply_blur(img):
blur_img = cv2.GaussianBlur(img, (3, 3), 0)
return blur_img
class MyTrainer(DefaultTrainer):
@classmethod
def build_train_loader(cls, cfg):
augmentations = [
T.RandomFlip(),
T.ColorTransform(apply_blur),
]
mapper = DatasetMapper(cfg, is_train=True, augmentations=augmentations)
return build_detection_train_loader(cfg, mapper=mapper)
```
I found [AugInput](https://detectron2.readthedocs.io/en/latest/modules/data_transforms.html#detectron2.data.transforms.AugInput), but it expects image, mask, etc. as input. Instead, is there something like the following exists?
```
def my_transform(img, mask):
# do something on image using mask and return image
return img
...
def build_train_loader(cls, cfg):
augmentations = [
T.RandomFlip(),
T.CustomTransform(my_transform),
]
...
```
Thank you very much
----------------------------
## Instructions To Reproduce the 🐛 Bug:
1. Full runnable code or full changes you made:
```
If making changes to the project itself, please use output of the following command:
git rev-parse HEAD; git diff
```
2. What exact command you run:
3. __Full logs__ or other relevant observations:
```
```
4. please simplify the steps as much as possible so they do not require additional resources to
run, such as a private dataset.
## Expected behavior:
If there are no obvious error in "full logs" provided above,
please tell us the expected behavior.
## Environment:
Provide your environment information using the following command:
```
wget -nc -q https://github.com/facebookresearch/detectron2/raw/main/detectron2/utils/collect_env.py && python collect_env.py
```
If your issue looks like an installation issue / environment issue,
please first try to solve it yourself with the instructions in
https://detectron2.readthedocs.io/tutorials/install.html#common-installation-issues
Contributor guide
Research direction
Start by reading the documented ColorTransform and AugInput APIs, then trace how DatasetMapper applies augmentations from build_train_loader. Compare the requested image-and-mask callable with the existing transform entry points; done would require a decided, supported approach for mask-aware augmentation and corresponding documentation or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- opencv, python
- Domain
- computer-vision
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100