google-research / google-research/perceiver-ar
Training on HuggingFace OpenWebText for GPT2 comparison
Open
- Dominant language
- Python
- Stars
- 257
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
On my fork I'm attempting to add in support for HuggingFace's OpenWebText dataset and their GPT2 tokenizer so that I can do a comparison against HF's GPT2-small. If you were willing, I'd love advice on setting up model params for self-supervised autoregressive NLP. Thanks!
Contributor guide
Research direction
The issue mentions a fork, HuggingFace's OpenWebText dataset, its GPT2 tokenizer, and comparison with GPT2-small, but names no repository files or tests. Start by reviewing the existing training entry points and model-parameter configuration, then define the dataset/tokenizer integration and comparison criteria before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100