JuliaText / JuliaText/TextModels.jl

Implementation of ROBERT on top of Transformers and Flux

Open
#15 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
30
Forks
14
PR merge metrics
No merged PRs in 30d

Description

- This implementation is the same as `Transformers.Bert` with a tiny embeddings tweaks.
- [RoBERTa](https://arxiv.org/abs/1907.11692) has the same architecture as BERT, but uses a byte-level BPE(implemented in `BPE.jl`) as a tokenizer (same as GPT-2) and uses a different pre-training scheme.
- RoBERTa doesn’t have token_type_ids, you don’t need to indicate which token belongs to which segment. Just separate your segments with the separation (or ``)

we can also wrapper [Camembert](https://camembert-model.fr/) (or the french version of BERT) around RoBERT.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.