facebookresearch / facebookresearch/mmf

Unable to reproduce model performance using pretrained models

Open
#952 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.6k
Forks
939
PR merge metrics
No merged PRs in 30d

Description

## ❓ Questions and Help
Hi everyone,

I have been trying to reproduce the model performance stated on the research paper but to no avail. Hence, I would like to clarify whether my command is correct.

For starters, I have downloaded the dataset from https://hatefulmemeschallenge.com/. Subsequently, I ran the following code (for Visual BERT COCO) based my understanding of the README.md:
```python
mmf_run dataset=hateful_memes \
model=visual_bert \
config=projects/hateful_memes/configs/visual_bert/from_coco.yaml \
checkpoint.resume_file=visual_bert.finetuned.hateful_memes.from_coco \
checkpoint.resume_pretrained=False \
run_type=val
```

Using this, I achieved an underwhelming result of 0.5925 for accuracy and 0.4806 ROC AUC score. This seems to be far below the performance reported in the paper...

P.S. I have also seen [Issue 926](https://github.com/facebookresearch/mmf/issues/926). Does this means the trained model checkpoint has not been updated and we should train the model from scratch?

Contributor guide

Open the contributing guide

Research direction

Start with README.md and the provided mmf_run command, then inspect projects/hateful_memes/configs/visual_bert/from_coco.yaml and the pretrained checkpoint reference. Compare the reported validation metrics with Issue 926; done means confirming whether the checkpoint and command reproduce the paper’s performance or documenting the discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.