Lightning-AI / Lightning-AI/pytorch-lightning
chunkable datasets and dataloaders
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Description & Motivation
Current large model training requires a huge number of traning samples, so that the traditional Mapped dataloaders are failed to load the training data because of limited memory. so I think there can be a chunkable datasets and dataloaders available. so that we can use the Mapped logic to load the first subset of training data in the dataset and dataloader 1 for training and the loading the second subset of training data for prepare.
### Pitch
I had seen the CombinedLoader in lighting, and I did not find too much documents and examples about it, It seems that it can not resolve the requirement about **"loading large traning data in memory chunk by chunk"**. If there had been some resolution, please give me some help, thanks!
### Alternatives
_No response_
### Additional context
_No response_
cc @lantiga @borda @tchaton
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the CombinedLoader documentation and examples, then compare its behavior with the mapped dataloaders described in the issue. The work is complete when a documented approach supports loading large training data chunk by chunk without requiring the full dataset in memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100