deepspeedai / deepspeedai/DeepSpeed
[REQUEST] Data Setup Instructions for BERT Pre-Training
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Is your feature request related to a problem? Please describe.
At https://www.deepspeed.ai/tutorials/bert-pretraining/#training-data-setup, DeepSpeed team noted that downloading and pre-processing instructions are coming soon. However, I haven't found any new instructions uploaded to the website for about two or three months. I have no idea on how to contact the team via e-mail, thus I'm here looking for some help. Thanks a lot!
Describe the solution you'd like
Could you please post instructions for the pre-processing stage in BERT pre-training to the website? Or at least give some references on how to pre-process the dataset?
Describe alternatives you've considered
N/A
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the BERT pre-training tutorial's training-data-setup section at https://www.deepspeed.ai/tutorials/bert-pretraining/#training-data-setup and determine what downloading and preprocessing guidance is missing. Done means the website contains usable preprocessing instructions or references for the dataset.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100