facebookresearch / facebookresearch/perception_models

Reproduce the video zero-shot evaluation results

Open
#114 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.4k
Forks
162
PR merge metrics
No merged PRs in 30d

Description

Hi, thanks for sharing the great work.

I noticed that the paper mentions:
> extending the CLIPBench zero-shot evaluation to include video datasets such as MSR-VTT and Kinetics.

While reproducing the video zero-shot evaluation results from the paper, I found that the source of these evaluation data is unclear, and benchmark test results from different sources cannot be aligned. Could you please provide download links for the video evaluation data (e.g., from the original website or Hugging Face) and details on the specific val/test splits to help reproduce the video evaluation results reported in the paper?

> from
> https://github.com/facebookresearch/perception_models/blob/3e352cca660658d4b5c90f42a7808b11469e4c66/apps/pe/clip_benchmark/datasets/builder.py#L19
> https://github.com/facebookresearch/perception_models/blob/3e352cca660658d4b5c90f42a7808b11469e4c66/apps/pe/clip_benchmark/datasets/builder.py#L27

> https://github.com/facebookresearch/perception_models/blob/3e352cca660658d4b5c90f42a7808b11469e4c66/apps/pe/clip_benchmark/datasets/builder.py#L23

> https://github.com/facebookresearch/perception_models/blob/3e352cca660658d4b5c90f42a7808b11469e4c66/apps/pe/clip_benchmark/datasets/builder.py#L27

> https://github.com/facebookresearch/perception_models/blob/3e352cca660658d4b5c90f42a7808b11469e4c66/apps/pe/clip_benchmark/datasets/builder.py#L34

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.