DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA3
Question about Differential Frame Pruner
- Dominant language
- Jupyter Notebook
- Stars
- 1.2k
- Forks
- 89
- PR merge metrics
- No merged PRs in 30d
Description
def _get_compression_mask(
self,
pixel_values: torch.FloatTensor,
batched_num_patches: torch.LongTensor,
grid_sizes: torch.LongTensor,
merge_sizes: torch.LongTensor,
modals: List[str],
threshold: float = 0.1,
min_tokens: int = 1,
) -> torch.BoolTensor:
batched_images = pixel_values.split(grid_sizes.prod(dim=1).tolist(), dim=0)
compression_masks = []
for images, num_patches, grid_size, merge_size, modal in zip(
batched_images, batched_num_patches, grid_sizes, merge_sizes, modals
):
t, h, w = grid_size
if modal == "image" or (modal == "video" and t == 1):
compression_masks.append(torch.ones((num_patches,), dtype=torch.bool, device=images.device))
elif modal == "video":
# NOTE: video token compressor
images = images.view(t, (h // merge_size) * (w // merge_size), -1)
pixel_diff = images[1:] - images[:-1]
pixel_diff = torch.abs(pixel_diff).mean(dim=-1) * 255
pixel_diff = torch.cat([torch.full_like(pixel_diff[0:1], threshold + 1), pixel_diff], dim=0)
mask = pixel_diff > threshold
padding_ids = torch.nonzero(mask.sum(dim=1) < min_tokens)[:, 0]
# mask[padding_ids, torch.randperm(min_tokens)] = 1
mask[padding_ids, :min_tokens] = 1
compression_masks.append(mask.flatten())
else:
# in case of psuedo image
compression_masks.append(torch.ones((0,), dtype=torch.bool, device=images.device))
return torch.cat(compression_masks)
it seems the value of pixel_diff is range[0, 255]
but threshold is 0.1
mask = pixel_diff > threshold
almost all pixel_diff is greater than threshold, so mask should be always 1
is that true?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.