allenai / allenai/bi-att-flow

f1 and em score is less than official result by 0.3%

Open
#60 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
668
PR merge metrics
No merged PRs in 30d

Description

I followed the exact instructions in the 'readme.md' file and started training my model with the following command:
` python -m basic.cli --mode train --noload --len_opt --cluster --batch_size 50`
After 18K steps I used the following command to test the model
`python squad/evaluate-v1.1.py $HOME/data/squad/dev-v1.1.json out/basic/00/answer/test-####.json`
and then I got f1=74.982, exact_match=64.90.
The scores for a single model in the original paper are em=68.0 and f1=77.3. And mine are 0.3 % point lower than those.
Because the codes are provided by the official group , the hyper parameters are exactly the same except the batch_size which won't affect the models' performance critically. The only reason I can think of is the different initial value.
Has anyone done the same work as I do ? Or can anyone provides other ideas?
Thanks a lot!!!!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.