codertimo / codertimo/BERT-pytorch

Default model sizes are much smaller than BERT base

Open
#81 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.5k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

The base BERT model in https://arxiv.org/pdf/1810.04805.pdf uses 768 hidden features, 12 layers, 12 heads (which are also the defaults in `bert.py`), while the default configuration in the argparser of `__main__.py` uses 256/8/8. Would it make sense to align the example script with the paper? I spent quite a while puzzling over my low GPU utilization with the default configuration. Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.