jonathanbratt / jonathanbratt/RBERT

Load BERT-esque checkpoints in pytorch formats

Open
#20 3 comments 3 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
R
Stars
166
Forks
19
PR merge metrics
No merged PRs in 30d

Description

The original BERT checkpoints released by Google are in a TensorFlow format.
It seems that most of the related work done by other teams is in the [PyTorch](https://github.com/huggingface/pytorch-transformers) implementation.
In particular, pre-trained models such as [RoBERTa](https://arxiv.org/abs/1907.11692) and [DistilBERT](https://medium.com/huggingface/distilbert-8cf3380435b5) have been released for PyTorch.

Many of these models are compatible with the BERT architecture, though possibly with different parameters or vocabularies. It would be great to be able to easily load these into RBERT.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing RBERT's existing checkpoint-loading path and the linked PyTorch implementation, then compare the TensorFlow BERT checkpoints with PyTorch models such as RoBERTa and DistilBERT. Done means the project can clearly and reliably load the compatible PyTorch-format models, including differences in parameters and vocabularies.

Written by the indexing model from the issue text.

Assessment

Tech stack
pytorch, r, tensorflow
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.