bytedance / bytedance/LPCD

Padded patches are aggregated as user 0 and timeslot 0

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Summary

Padded patch slots are treated as real user-0 and timeslot-0 patches during entity aggregation. This happens because `patch_u_t_masks` uses zero for unfilled slots, while zero is also a valid anchor-user and first-timeslot ID, and `collect_entity_patch_embs` does not receive the existing `patch_valid_mask`.

This is still present on `main` at `a0b47382f08b108a17a39b343c4e58dbf2852db3`.

## Code path

Both `model/ours/ACMIL_v2.py` and `model/ours/patchMIL_v20.py`:

1. allocate `patch_u_t_masks` with zeros;
2. populate user/time IDs only for valid actions;
3. derive `patch_valid_mask` from `patch_action_valid_masks`;
4. call `collect_entity_patch_embs(patch_u_t_masks[..., 0 or 1], ...)` without passing validity.

`model_utils.collect_entity_patch_embs` then counts and marks every `[B, P]` slot as valid. In a batch where rooms have different patch counts, trailing slots for shorter rooms therefore enter entity 0. `model_utils.mask_generation` explicitly documents user ID 0 as the anchor, and timeslot 0 is also a real bucket.

## Minimal reproduction

```python
entity_idx = torch.tensor([[1, 0]]) # second slot is padding represented as zero
patch_embs = torch.tensor([[[9.], [1.], [0.]]]) # CLS, real patch, padded patch
_, mask = collect_entity_patch_embs(entity_idx, 2, patch_embs)
print(mask[0, 0].sum())
```

Actual: `1` — entity 0 is reported to contain a patch even though the only real patch belongs to entity 1.

Expected: `0` when the second slot's `patch_valid_mask` is false.

## Impact

For mixed-length batches, the anchor-user and first-timeslot pooling masks include padded zero embeddings. Their attention normalization and global representations therefore depend on how many patches other rooms in the same batch contain, contaminating logits and training gradients with batch-composition artifacts.

## Suggested fix

Allow `collect_entity_patch_embs` to filter by the caller's `patch_valid_mask`, pass it from every ACMIL/patchMIL call site, and add a regression test containing a valid entity-0 patch plus an invalid padded slot.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with collect_entity_patch_embs in model_utils and trace the call sites in model/ours/ACMIL_v2.py and model/ours/patchMIL_v20.py. Reproduce the mixed-length batch case from the issue, then add a regression test covering a valid entity-0 patch and an invalid padded slot. Done means padded slots are excluded from entity masks and the regression test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.