facebookresearch / facebookresearch/sam3

`input_points` shape mismatch when batch has mixed point prompts (BS>1)

Open
#458 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.7k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

## Problem: `input_points` shape mismatch when batch has mixed point prompts (BS>1)

In `collate.py` (`collate_fn_api`), `input_points_embedding_dim` is set to **257**. During collation, when `q.input_points` exists, we append `q.input_points.squeeze(0)`; otherwise we append an empty tensor with shape `(0, input_points_embedding_dim)`.

```python
if q.input_points is not None:
stages[stage_id].input_points.append(
q.input_points.squeeze(0) # Strip a trivial batch index
)
stages[stage_id].input_points_mask.append(
torch.zeros(q.input_points.shape[1])
)
else:
stages[stage_id].input_points.append(
torch.empty(0, input_points_embedding_dim)
)
stages[stage_id].input_points_mask.append(torch.empty(0))
```
When batch size > 1 and the batch contains mixed samples (some have point prompts, some don't):

- Samples with point prompts: stages[stage_id].input_points element shape becomes [N, 3]
- Samples without point prompts: default empty tensor shape is [0, 257] (because input_points_embedding_dim=257)

Later, in function `convert_my_tensors`, these tensors cannot be stacked because the dimension is inconsistent (3 vs 257). Should `input_points_embedding_dim` be 3 instead of 257? Any guidance on what the intended representation is at this stage?

Contributor guide

Open the contributing guide

Research direction

Start in collate.py at collate_fn_api and trace the input_points values into convert_my_tensors. Compare the shapes for samples with and without point prompts in a batch larger than one, then confirm the intended representation from the surrounding model inputs. Done means mixed point-prompt batches collate and stack without a dimension mismatch, with the representation documented or covered by an applicable test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.