facebookresearch / facebookresearch/sam3
[SAM 3.1 Multiplex] Explicit forward tracking bound can enter an empty grounding chunk
- Dominant language
- Python
- Stars
- 11.7k
- Forks
- 1.8k
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
In the SAM 3.1 Multiplex video path, an explicit forward `max_frame_num_to_track` is interpreted with two different boundary conventions:
- `_get_processing_order` treats it as an inclusive offset and schedules `N + 1` frames.
- `forward_video_grounding_batched_multigpu` uses it to form an exclusive right bound and prepares only `N` grounding inputs.
When propagation reaches the extra final frame, the grounding chunk is empty (with grounding batch size 1), and `_batch_find_inputs` raises `IndexError: list index out of range`.
I reproduced this against `main` at [`660a5e9e1b8b4c02c0ad97229b88a09a6e4ff5b7`](https://github.com/facebookresearch/sam3/commit/660a5e9e1b8b4c02c0ad97229b88a09a6e4ff5b7) with the SAM 3.1 Multiplex checkpoint.
## Reproduction
1. Initialize a SAM 3.1 Multiplex video session containing at least 16 frames.
2. Set `predictor.model.batched_grounding_batch_size = 1`.
3. Add a text prompt on frame 0.
4. Run forward propagation with an explicit bound smaller than the remaining video:
```python
request = {
type: propagate_in_video,
session_id: session_id,
propagation_direction: forward,
start_frame_index: 0,
max_frame_num_to_track: 15,
}
for response in predictor.handle_stream_request(request):
pass
```
The progress reaches `15/16` and then fails:
```text
Running full VG propagation (reverse=False).
propagate_in_video: 94%|...| 15/16
File sam3_multiplex_tracking.py, line 351, in propagate_in_video
out = self._run_single_frame_inference(...)
File sam3_multiplex_detector.py, line 750, in forward_video_grounding_batched_multigpu
chunk_outputs = self._process_grounding_chunk_batched(...)
File sam3_multiplex_detector.py, line 809, in _process_grounding_chunk_batched
batched_find_input = self._batch_find_inputs(...)
File sam3_multiplex_detector.py, line 140, in _batch_find_inputs
device = chunk_find_inputs[0].img_ids.device
IndexError: list index out of range
```
## Cause
For `start_frame_idx=0` and `max_frame_num_to_track=15`, the outer tracking loop schedules frames 0 through 15 (16 frames):
https://github.com/facebookresearch/sam3/blob/660a5e9e1b8b4c02c0ad97229b88a09a6e4ff5b7/sam3/model/sam3_multiplex_tracking.py#L295-L297
```python
end_frame_idx = start_frame_idx + max_frame_num_to_track
processing_order = range(start_frame_idx, end_frame_idx + 1)
```
However, grounding defines `valid_frame_end` as an exclusive bound of 15:
https://github.com/facebookresearch/sam3/blob/660a5e9e1b8b4c02c0ad97229b88a09a6e4ff5b7/sam3/model/sam3_multiplex_detector.py#L727-L742
```python
valid_frame_end = propagate_in_video_start_frame_idx + max_frame_num_to_track
chunk_end = min(chunk_start + batch_size, valid_frame_end)
```
On outer-loop frame 15 with `batch_size=1`, both `chunk_start` and `chunk_end` are 15. This creates an empty `chunk_find_inputs`, which is indexed unconditionally here:
https://github.com/facebookresearch/sam3/blob/660a5e9e1b8b4c02c0ad97229b88a09a6e4ff5b7/sam3/model/sam3_multiplex_detector.py#L133-L140
The mismatch exists regardless of batch size, although a larger batch can make the eventual failure appear at a different indexing point.
## Expected behavior
The outer propagation order and grounding range should use one consistent convention. Based on the parameter name and the default assignment `max_frame_num_to_track = num_frames`, I would expect a value of 15 to process exactly 15 frames, including the start frame (frames 0 through 14).
One possible forward-path correction is:
```python
end_frame_idx = start_frame_idx + max_frame_num_to_track - 1
end_frame_idx = min(end_frame_idx, num_frames - 1)
processing_order = range(start_frame_idx, end_frame_idx + 1)
```
The reverse path should be reviewed at the same time so both directions have documented, symmetric semantics. Merely ignoring the empty chunk would hide the boundary mismatch and can also leave results buffered by `hotstart_delay` instead of returning a complete propagation result.
## Current workaround
We wrap `_get_processing_order` so that only the outer loop receives `max_frame_num_to_track - 1`, while the original full value remains in `feature_cache[tracking_bounds]` for grounding. This prevents the extra iteration without shortening the grounding input range.
Contributor guide
Research direction
Start with _get_processing_order in sam3/model/sam3_multiplex_tracking.py and the bounds calculation in forward_video_grounding_batched_multigpu in sam3/model/sam3_multiplex_detector.py. Run the supplied 16-frame reproduction with grounding batch size 1, then verify that forward and reverse propagation use documented, consistent frame-count semantics without an empty grounding chunk or incomplete buffered results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100