facebookresearch / facebookresearch/sam3

[Potential bug] Object Pointer Token Not Matched to Best Mask in Multimask Mode for Sam2 Tracking

Open
#475 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.7k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

## Summary

When `multimask_output_in_sam=True` and `use_multimask_token_for_obj_ptr=True`, the object pointer token used for memory encoding is always taken from index 0 of the multimask tokens, regardless of which mask was selected as best (by highest IoU). This means the object pointer may encode information from a different mask than the one actually stored in spatial memory.

## Affected File

`sam3/model/sam3_tracker_base.py`, method `_forward_sam_heads`

## Code in Question

https://github.com/facebookresearch/sam3/blob/f6e51f59500a87c576c2df2323ce56b9fd7a12de/sam3/model/sam3_tracker_base.py#L359-L369

## Why This Is a Bug

Each mask token in the SAM decoder directly generates its corresponding mask via a hypernetwork MLP:

https://github.com/facebookresearch/sam3/blob/f6e51f59500a87c576c2df2323ce56b9fd7a12de/sam3/sam/mask_decoder.py#L226-L234

The token at index `i` is semantically tied to mask `i` — they share the same learned representation from the TwoWayTransformer. When `multimask_output=True`, the best mask is selected by `argmax(ious)` and may be at index 1 or 2. But `sam_output_token` is always taken from index 0, creating an inconsistency:

- **Spatial memory** (`maskmem_features`) is encoded from the best mask (index `best_iou_inds`)
- **Object pointer** (`obj_ptr`) is encoded from token index 0, which may correspond to a *different* mask

This means the two memory components — spatial and pointer — may represent different mask hypotheses for the same frame, which can degrade tracking quality.

## Configuration Where This Occurs

In the default SAM3 model builder (`sam3/model_builder.py`):

```python
multimask_output_in_sam=True,
multimask_output_for_tracking=True,
multimask_min_pt_num=0,
multimask_max_pt_num=1,
```

With `use_multimask_token_for_obj_ptr=True` in the tracker, `sam_output_tokens` has shape `(B, 3, C)` when `multimask_output=True`. The fix should select the token matching the best mask.

## Suggested Fix

```python
sam_output_token = sam_output_tokens[:, 0]

if multimask_output:
best_iou_inds = torch.argmax(ious, dim=-1)
batch_inds = torch.arange(B, device=ious.device)
low_res_masks = low_res_multimasks[batch_inds, best_iou_inds, :]
high_res_masks = high_res_multimasks[batch_inds, best_iou_inds, :]
# Fix: select token matching the best mask
if sam_output_tokens.size(1) > 1:
sam_output_token = sam_output_tokens[batch_inds, best_iou_inds]

obj_ptr = self.obj_ptr_proj(sam_output_token)
```

Contributor guide

Open the contributing guide

Research direction

Start in sam3/model/sam3_tracker_base.py at _forward_sam_heads, then read the referenced mask-decoder token-to-mask logic and the multimask settings in sam3/model_builder.py. Confirm how the highest-IoU mask is selected and verify that the corresponding token is used for the object pointer; the issue is done when both memory components represent the selected mask consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.