facebookresearch / facebookresearch/sam2

[Question] Training the Memory Encoder

Open
#712 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
19.9k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Description

I read the paper and was a bit confused about how they train the memory encoder. T

hey say they sample 8 frames. Does that mean the memory encoder only ever has 8 frames maximum? What if the object goes out of frame for longer than 8 frames?

Also, since each prediction relies on the memory bank from the previous prediction, does this mean this is a recurrent network architecture, or is there some way to do teacher forcing by feeding in the ground truth?

If someone could help answer these questions or explain how the memory encoder is trainined, that would be super helpful for my understanding!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.