Can't Reproduce Zero Shot Performance MSRVTT and LSMDC with Intervid-10m-FLT Checkpoint

Open
#139 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python, pytorch
Domain
machine-learning, testing-qa

Research direction

Start with tasks/retrieval.py and the referenced config.py, then inspect how the ViClip-InternVid-10M-FLT.pth checkpoint is loaded and how MSRVTT and LSMDC retrieval metrics are evaluated. Re-run the provided torchrun command and compare both datasets' results with the paper's reported values. Done means identifying and documenting the source of the discrepancy or confirming the evaluation setup.

Written by the indexing model from the issue text.

Description

Dear Authors,
I am trying to reproduce Zeroshot performance with the checkpoint ViCLIP-L-14 InternVid-10M-FLT .
However, the performance is different from reported numbers in the paper. Here are the results I obtain:

MSRVTT:
txt_r1 txt_r5 txt_r10 txt_r_mean img_r1 img_r5 img_r10 img_r_mean r_mean
msrvtt_1k_test/ 38.9 62.2 74.0 58.37 39.4 61.9 73.0 58.10 58.23
msrvtt_1k_test_emb/ 39.0 62.2 73.3 58.17 39.1 63.2 73.9 58.73 58.45

LSMDC:

txt_r1 txt_r5 txt_r10 txt_r_mean img_r1 img_r5 img_r10 img_r_mean r_mean
test/ 15.2 29.0 35.6 26.6 17.8 32.1 40.1 30.00 28.30
test_emb/ 15.8 29.1 36.7 27.2 18.5 32.7 40.8 30.67 28.93

Here is the script that i run to obtain the performances:

source /ibex/user/thokerfm/anaconda3/bin/activate viclip
export PYTHONPATH=.

MASTER_NODE=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1)
MASTER_PORT=$((RANDOM % (65535 - 1024 + 1) + 1024))

echo $MASTER_NODE
echo $MASTER_PORT

OUTPUT_DIR='expirements_zero_shot/ViClip-InternVid-10M-FLT/lsmdc/'

OMP_NUM_THREADS=1
torchrun --rdzv_endpoint=${MASTER_NODE}:${MASTER_PORT}
--nnodes=1
--nproc_per_node=4
--rdzv_backend=c10d
tasks/retrieval.py
$(dirname $0)/config.py
wandb.enable False
train_corpus viclip
evaluate True
output_dir ${OUTPUT_DIR}
model.vision_encoder.pretrained 'CLIP-ViT-L/14'
model.text_encoder.pretrained 'CLIP-ViT-L/14'
pretrained_path pretrained_viclip_models/ViClip-InternVid-10M-FLT.pth

Dominant language
Python
Stars
2.4k
Forks
160
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenGVLab/InternVideo

All issues in OpenGVLab/InternVideo

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.