Padded patches are aggregated as user 0 and timeslot 0
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Padded patch slots are treated as real user-0 and timeslot-0 patches during entity aggregation. This happens because `patch_u_t_masks` uses zero for unfilled slots, while zero is also a valid anchor-user and first-timeslot ID, and `collect_entity_patch_embs` does not receive the existing `patch_valid_mask`.
This is still present on `main` at `a0b47382f08b108a17a39b343c4e58dbf2852db3`.
## Code path
Both `model/ours/ACMIL_v2.py` and `model/ours/patchMIL_v20.py`:
1. allocate `patch_u_t_masks` with zeros;
2. populate user/time IDs only for valid actions;
3. derive `patch_valid_mask` from `patch_action_valid_masks`;
4. call `collect_entity_patch_embs(patch_u_t_masks[..., 0 or 1], ...)` without passing validity.
`model_utils.collect_entity_patch_embs` then counts and marks every `[B, P]` slot as valid. In a batch where rooms have different patch counts, trailing slots for shorter rooms therefore enter entity 0. `model_utils.mask_generation` explicitly documents user ID 0 as the anchor, and timeslot 0 is also a real bucket.
## Minimal reproduction
```python
entity_idx = torch.tensor([[1, 0]]) # second slot is padding represented as zero
patch_embs = torch.tensor([[[9.], [1.], [0.]]]) # CLS, real patch, padded patch
_, mask = collect_entity_patch_embs(entity_idx, 2, patch_embs)
print(mask[0, 0].sum())
```
Actual: `1` — entity 0 is reported to contain a patch even though the only real patch belongs to entity 1.
Expected: `0` when the second slot's `patch_valid_mask` is false.
## Impact
For mixed-length batches, the anchor-user and first-timeslot pooling masks include padded zero embeddings. Their attention normalization and global representations therefore depend on how many patches other rooms in the same batch contain, contaminating logits and training gradients with batch-composition artifacts.
## Suggested fix
Allow `collect_entity_patch_embs` to filter by the caller's `patch_valid_mask`, pass it from every ACMIL/patchMIL call site, and add a regression test containing a valid entity-0 patch plus an invalid padded slot.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with collect_entity_patch_embs in model_utils and trace the call sites in model/ours/ACMIL_v2.py and model/ours/patchMIL_v20.py. Reproduce the mixed-length batch case from the issue, then add a regression test covering a valid entity-0 patch and an invalid padded slot. Done means padded slots are excluded from entity masks and the regression test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100