Question on the prompt information addition in the decoder

@nikhilaravi @HannaMao

I have a question on the following part of the paper. Could you please help with the following information?

"_To ensure the decoder has access to critical geometric information the positional encodings are added to the image embedding whenever they participate in an attention layer. Additionally, the entire original prompt tokens (including their positional encodings) are re-added to the updated tokens whenever they participate in an attention layer. This allows for a strong dependence on both the prompt token’s geometric location and type._"

It says that the entire original prompt tokens (_including their positional encoding_) are added to each out coming tokens of the decoder layers. What are the position encodings of the prompts? For e.g., for the dense prompts, i.e., for the masks, according to the info in the paper, they are downsampled with strided convolutions to match the image dimension and added to the image elementwise, what is the positional encoding of the mask? Similarly, how is it for the point prompts? I seem to have missed this information in the paper.

Also, just to confirm, the _original_ prompt information is not just added in the beginning but added to every layer of the decoder's output, is that correct? However, from the `https://github.com/facebookresearch/segment-anything/blob/main/segment_anything/modeling/transformer.py` code, it looks like the original, i.e, the initial queries are not added every time but the last iteration's updated queries are added each time. Could you please clarify what the _original_ query refers to?

_A second question_, in this part of the code in the mask decoder
`# Expand per-image data in batch direction to be per-mask

src = torch.repeat_interleave(image_embeddings, tokens.shape[0], dim=0)
src = src + dense_prompt_embeddings
pos_src = torch.repeat_interleave(image_pe, tokens.shape[0], dim=0)`

If I understand correctly, the image embedding is repeated for the total number of tokens in the transformer, i.e., each token in the transformer decoder has the same image embedding and its corresponding information, is that correct?

Thanks in advance.

Start by reading segment_anything/modeling/transformer.py and the mask decoder code referenced in the issue, then compare those paths with the paper passage about prompt information. Trace how image embeddings, prompt tokens, positional encodings, and updated queries are passed through decoder layers. Done means documenting a clear explanation of the behavior and resolving the repeated-image-embedding question.

Written by the indexing model from the issue text.