Lightning-AI / Lightning-AI/pytorch-lightning

chunkable datasets and dataloaders

Open
#20,576 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

data handling feature
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Description & Motivation

Current large model training requires a huge number of traning samples, so that the traditional Mapped dataloaders are failed to load the training data because of limited memory. so I think there can be a chunkable datasets and dataloaders available. so that we can use the Mapped logic to load the first subset of training data in the dataset and dataloader 1 for training and the loading the second subset of training data for prepare.

### Pitch

I had seen the CombinedLoader in lighting, and I did not find too much documents and examples about it, It seems that it can not resolve the requirement about **"loading large traning data in memory chunk by chunk"**. If there had been some resolution, please give me some help, thanks!

### Alternatives

_No response_

### Additional context

_No response_

cc @lantiga @borda @tchaton

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the CombinedLoader documentation and examples, then compare its behavior with the mapped dataloaders described in the issue. The work is complete when a documented approach supports loading large training data chunk by chunk without requiring the full dataset in memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.