Lightning-AI / Lightning-AI/litData
Cycle option for StreamingDataLoader
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 614
- Forks
- 106
- Avg merge
- 15h 8m
- Merged PRs (30d)
- 22
Description
## 🚀 Feature
A function or an argument in StreamingDataLoader to cycle the passed in StreamingDataset.
### Motivation
Many training scenarios in CV involve training models with multiple epochs, while wanting to control the exact number of steps being trained, independent of the underlying dataset size. E.g., given a CombinedStreamingDataset of some length, restart its iterations when it is exhausted.
### Pitch
I'm not quite sure how this should be done - maybe in __iter__ method of StreamingDataLoader, we can catch the final iteration and restart it?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the StreamingDataLoader __iter__ method and the behavior of StreamingDataset and CombinedStreamingDataset when iteration is exhausted. Determine how cycling should be exposed and how it should interact with controlling the exact number of training steps. Done means the loader can restart iteration after exhaustion for the requested training scenario, with tests covering the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100