facebookresearch / facebookresearch/sam2

Loading embeddings to speed up video predictions

Open
#565 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
19.9k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Description

For the SAM2 image predictor, I created a custom initialization of the image predictor which can save/load the image embeddings to speed up the process when those images are called for prediction again.

I'm interested in doing something similar for the SAM2 video predictor where I preload the embeddings for all frames. Is this (or something similar) possible for the video predictor as the embeddings/memory attention seem to differ from the image predictor.

I'm also having trouble getting the model to initialize correctly with a custom initialization using something like this:

```
class MySamVideo(SAM2VideoPredictor):
def __init__(self, model, **kwargs):
# Extract required attributes from `model` if available
extracted_args = {
"image_encoder": getattr(model, "image_encoder", None),
"memory_attention": getattr(model, "memory_attention", None),
"memory_encoder": getattr(model, "memory_encoder", None)
}

super().__init__(**extracted_args, **kwargs)
self.model = model

def save_image_embedding(self, path):
# Save existing embeddings

def load_image_embedding(self, path):
# Check if embedding exists & load

```

```
# called like this
sam = build_sam2_video_predictor(model_cfg, model_path, device=device)
sam.to(device=device)
model = MySamVideo(sam)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.