Memory encoder and mask attention for multiple objects in the same batch run

Dear Developers,

I have one theoretical question regarding the memory mechanism of the model. Referring to the SAM2Train.prepare_promt_inputs, I see that each object in the scene in the BatchedVideoDatapoint is processed as a batch, resulting in individual prompts for each object and also individual memory mechanisms for each object. Basically, for one batch, only the mask of a given object is encoded in the memory encoder and used in the memory attention for the next frame. I aim to estimate N object masks and use the memory mechanism you developed with slight changes. Have you tried to do it in such a sense that you encode the entire segmentation mask(1xHxW, including all the object masks) directly and then use the encoded mask in the memory attention?

Best