facebookresearch / facebookresearch/sam2
Change image resolution
- Dominant language
- Jupyter Notebook
- Stars
- 19.9k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Description
Similar to SAM 1, SAM 2 was trained on 1024x1024 images. I'm wondering whether it's possible to adapt SAM 2 to 512x512 images without resizing those images (since that takes a lot of time). This was possible in the old SAM 1 by resizing the positional embeddings, but I'm not sure how to go about it in this new repository. After changing `image_size` in the config, the only thing that seemed to cause an error was
```
feats = [
feat.permute(1, 2, 0).view(1, -1, *feat_size)
for feat, feat_size in zip(vision_feats[::-1], self._bb_feat_sizes[::-1])
][::-1]
```
I know SAM 1 uses a typical VIT, but SAM 2 uses hiera (which I'm not very familiar with). I changed the features in `SAM2ImagePredictor` from:
```
self._bb_feat_sizes = [
(256, 256),
(128, 128),
(64, 64),
]
```
to:
```
self._bb_feat_sizes = [
(128, 128),
(64, 64),
(32, 32),
]
```
and no errors were outputted, but I am not sure if the segmentation is working properly. Would appreciate feedback on this approach or any alternative methods!
Contributor guide
Research direction
Start with the SAM2ImagePredictor feature-size calculation and the configuration's image_size setting. Run inference on 512x512 images and verify that segmentation remains correct without resizing; the work is done when the predictor handles that resolution reliably and the feature reshaping matches the Hiera outputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100