codertimo / codertimo/BERT-pytorch
Default model sizes are much smaller than BERT base
Open
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
The base BERT model in https://arxiv.org/pdf/1810.04805.pdf uses 768 hidden features, 12 layers, 12 heads (which are also the defaults in `bert.py`), while the default configuration in the argparser of `__main__.py` uses 256/8/8. Would it make sense to align the example script with the paper? I spent quite a while puzzling over my low GPU utilization with the default configuration. Thanks!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.