f1 and em score is less than official result by 0.3%
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 668
- PR merge metrics
- No merged PRs in 30d
Description
I followed the exact instructions in the 'readme.md' file and started training my model with the following command:
` python -m basic.cli --mode train --noload --len_opt --cluster --batch_size 50`
After 18K steps I used the following command to test the model
`python squad/evaluate-v1.1.py $HOME/data/squad/dev-v1.1.json out/basic/00/answer/test-####.json`
and then I got f1=74.982, exact_match=64.90.
The scores for a single model in the original paper are em=68.0 and f1=77.3. And mine are 0.3 % point lower than those.
Because the codes are provided by the official group , the hyper parameters are exactly the same except the batch_size which won't affect the models' performance critically. The only reason I can think of is the different initial value.
Has anyone done the same work as I do ? Or can anyone provides other ideas?
Thanks a lot!!!!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.