Lightning-AI / Lightning-AI/litgpt
Is there any best practice for using litdata to load custom data for pretraining?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
Is there any best practice for using litdata to load custom data for pretraining? I found that TextFiles.py and prepare_slimpajama.py have similar data preprocessing methods. The difference between them is in the tokenizer. One is tokenizer during preprocessing, and the other is tokenizer during training？Why TextFiles are not suitable for handling large amounts of data？
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing TextFiles.py and prepare_slimpajama.py, focusing on where tokenization occurs and how each preprocessing path handles data volume. Use the issue's large-data question to determine what guidance is needed; done means documenting a clear recommended litdata workflow for custom pretraining data and explaining the tradeoffs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100