jonathanbratt / jonathanbratt/RBERT
Load BERT-esque checkpoints in pytorch formats
- Dominant language
- R
- Stars
- 166
- Forks
- 19
- PR merge metrics
- No merged PRs in 30d
Description
The original BERT checkpoints released by Google are in a TensorFlow format.
It seems that most of the related work done by other teams is in the [PyTorch](https://github.com/huggingface/pytorch-transformers) implementation.
In particular, pre-trained models such as [RoBERTa](https://arxiv.org/abs/1907.11692) and [DistilBERT](https://medium.com/huggingface/distilbert-8cf3380435b5) have been released for PyTorch.
Many of these models are compatible with the BERT architecture, though possibly with different parameters or vocabularies. It would be great to be able to easily load these into RBERT.
Contributor guide
Research direction
Start by reviewing RBERT's existing checkpoint-loading path and the linked PyTorch implementation, then compare the TensorFlow BERT checkpoints with PyTorch models such as RoBERTa and DistilBERT. Done means the project can clearly and reliably load the compatible PyTorch-format models, including differences in parameters and vocabularies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pytorch, r, tensorflow
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100