facebookresearch / facebookresearch/sam3
[Potential bug] Object Pointer Token Not Matched to Best Mask in Multimask Mode for Sam2 Tracking
- Dominant language
- Python
- Stars
- 11.7k
- Forks
- 1.8k
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
When `multimask_output_in_sam=True` and `use_multimask_token_for_obj_ptr=True`, the object pointer token used for memory encoding is always taken from index 0 of the multimask tokens, regardless of which mask was selected as best (by highest IoU). This means the object pointer may encode information from a different mask than the one actually stored in spatial memory.
## Affected File
`sam3/model/sam3_tracker_base.py`, method `_forward_sam_heads`
## Code in Question
https://github.com/facebookresearch/sam3/blob/f6e51f59500a87c576c2df2323ce56b9fd7a12de/sam3/model/sam3_tracker_base.py#L359-L369
## Why This Is a Bug
Each mask token in the SAM decoder directly generates its corresponding mask via a hypernetwork MLP:
https://github.com/facebookresearch/sam3/blob/f6e51f59500a87c576c2df2323ce56b9fd7a12de/sam3/sam/mask_decoder.py#L226-L234
The token at index `i` is semantically tied to mask `i` — they share the same learned representation from the TwoWayTransformer. When `multimask_output=True`, the best mask is selected by `argmax(ious)` and may be at index 1 or 2. But `sam_output_token` is always taken from index 0, creating an inconsistency:
- **Spatial memory** (`maskmem_features`) is encoded from the best mask (index `best_iou_inds`)
- **Object pointer** (`obj_ptr`) is encoded from token index 0, which may correspond to a *different* mask
This means the two memory components — spatial and pointer — may represent different mask hypotheses for the same frame, which can degrade tracking quality.
## Configuration Where This Occurs
In the default SAM3 model builder (`sam3/model_builder.py`):
```python
multimask_output_in_sam=True,
multimask_output_for_tracking=True,
multimask_min_pt_num=0,
multimask_max_pt_num=1,
```
With `use_multimask_token_for_obj_ptr=True` in the tracker, `sam_output_tokens` has shape `(B, 3, C)` when `multimask_output=True`. The fix should select the token matching the best mask.
## Suggested Fix
```python
sam_output_token = sam_output_tokens[:, 0]
if multimask_output:
best_iou_inds = torch.argmax(ious, dim=-1)
batch_inds = torch.arange(B, device=ious.device)
low_res_masks = low_res_multimasks[batch_inds, best_iou_inds, :]
high_res_masks = high_res_multimasks[batch_inds, best_iou_inds, :]
# Fix: select token matching the best mask
if sam_output_tokens.size(1) > 1:
sam_output_token = sam_output_tokens[batch_inds, best_iou_inds]
obj_ptr = self.obj_ptr_proj(sam_output_token)
```
Contributor guide
Research direction
Start in sam3/model/sam3_tracker_base.py at _forward_sam_heads, then read the referenced mask-decoder token-to-mask logic and the multimask settings in sam3/model_builder.py. Confirm how the highest-IoU mask is selected and verify that the corresponding token is used for the object pointer; the issue is done when both memory components represent the selected mask consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100