deepspeedai / deepspeedai/DeepSpeedExamples
DeepSpeedChat step 1 can not train on big datasets that need to be stored on disk and loaded by the dataloader
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
I'm trying to train 1.4millon tokens and loading all of that on each process is impossible. I'm wondering what people do to modify the code base to hold data on disk and load accordingly.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not name a file or test; start by locating the DeepSpeedChat step 1 dataloader and tracing how the training dataset is loaded. Determine how a dataset that remains on disk should be accessed across processes, then verify that step 1 can train on the reported large dataset without loading it all into each process.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100