DAMO-NLP-SG / DAMO-NLP-SG/VideoLLaMA3

Question about Differential Frame Pruner

Open
#28 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
89
PR merge metrics
No merged PRs in 30d

Description

def _get_compression_mask(
self,
pixel_values: torch.FloatTensor,
batched_num_patches: torch.LongTensor,
grid_sizes: torch.LongTensor,
merge_sizes: torch.LongTensor,
modals: List[str],
threshold: float = 0.1,
min_tokens: int = 1,
) -> torch.BoolTensor:
batched_images = pixel_values.split(grid_sizes.prod(dim=1).tolist(), dim=0)
compression_masks = []

for images, num_patches, grid_size, merge_size, modal in zip(
batched_images, batched_num_patches, grid_sizes, merge_sizes, modals
):
t, h, w = grid_size
if modal == "image" or (modal == "video" and t == 1):
compression_masks.append(torch.ones((num_patches,), dtype=torch.bool, device=images.device))

elif modal == "video":
# NOTE: video token compressor
images = images.view(t, (h // merge_size) * (w // merge_size), -1)

pixel_diff = images[1:] - images[:-1]
pixel_diff = torch.abs(pixel_diff).mean(dim=-1) * 255
pixel_diff = torch.cat([torch.full_like(pixel_diff[0:1], threshold + 1), pixel_diff], dim=0)
mask = pixel_diff > threshold
padding_ids = torch.nonzero(mask.sum(dim=1) < min_tokens)[:, 0]
# mask[padding_ids, torch.randperm(min_tokens)] = 1
mask[padding_ids, :min_tokens] = 1
compression_masks.append(mask.flatten())

else:
# in case of psuedo image
compression_masks.append(torch.ones((0,), dtype=torch.bool, device=images.device))

return torch.cat(compression_masks)

it seems the value of pixel_diff is range[0, 255]
but threshold is 0.1

mask = pixel_diff > threshold

almost all pixel_diff is greater than threshold, so mask should be always 1

is that true?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.