facebookresearch / facebookresearch/mmf
Unable to reproduce model performance using pretrained models
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 939
- PR merge metrics
- No merged PRs in 30d
Description
## ❓ Questions and Help
Hi everyone,
I have been trying to reproduce the model performance stated on the research paper but to no avail. Hence, I would like to clarify whether my command is correct.
For starters, I have downloaded the dataset from https://hatefulmemeschallenge.com/. Subsequently, I ran the following code (for Visual BERT COCO) based my understanding of the README.md:
```python
mmf_run dataset=hateful_memes \
model=visual_bert \
config=projects/hateful_memes/configs/visual_bert/from_coco.yaml \
checkpoint.resume_file=visual_bert.finetuned.hateful_memes.from_coco \
checkpoint.resume_pretrained=False \
run_type=val
```
Using this, I achieved an underwhelming result of 0.5925 for accuracy and 0.4806 ROC AUC score. This seems to be far below the performance reported in the paper...
P.S. I have also seen [Issue 926](https://github.com/facebookresearch/mmf/issues/926). Does this means the trained model checkpoint has not been updated and we should train the model from scratch?
Contributor guide
Research direction
Start with README.md and the provided mmf_run command, then inspect projects/hateful_memes/configs/visual_bert/from_coco.yaml and the pretrained checkpoint reference. Compare the reported validation metrics with Issue 926; done means confirming whether the checkpoint and command reproduce the paper’s performance or documenting the discrepancy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100