facebookresearch / facebookresearch/sam2

Change image resolution

Open
#138 22 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
19.9k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Description

Similar to SAM 1, SAM 2 was trained on 1024x1024 images. I'm wondering whether it's possible to adapt SAM 2 to 512x512 images without resizing those images (since that takes a lot of time). This was possible in the old SAM 1 by resizing the positional embeddings, but I'm not sure how to go about it in this new repository. After changing `image_size` in the config, the only thing that seemed to cause an error was
```
feats = [
feat.permute(1, 2, 0).view(1, -1, *feat_size)
for feat, feat_size in zip(vision_feats[::-1], self._bb_feat_sizes[::-1])
][::-1]
```

I know SAM 1 uses a typical VIT, but SAM 2 uses hiera (which I'm not very familiar with). I changed the features in `SAM2ImagePredictor` from:
```
self._bb_feat_sizes = [
(256, 256),
(128, 128),
(64, 64),
]
```
to:
```
self._bb_feat_sizes = [
(128, 128),
(64, 64),
(32, 32),
]
```

and no errors were outputted, but I am not sure if the segmentation is working properly. Would appreciate feedback on this approach or any alternative methods!

Contributor guide

Open the contributing guide

Research direction

Start with the SAM2ImagePredictor feature-size calculation and the configuration's image_size setting. Run inference on 512x512 images and verify that segmentation remains correct without resizing; the work is done when the predictor handles that resolution reliably and the feature reshaping matches the Hiera outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.