Lightning-AI / Lightning-AI/litData
Support `StreamingDataLoader` passed to `map`
- Dominant language
- Python
- Stars
- 614
- Forks
- 106
- Avg merge
- 15h 8m
- Merged PRs (30d)
- 22
Description
## 🚀 Feature
It would be great to be able to pass a StreamingDataLoader to map. When experimenting with CLIP embeddings I've found that I needed to use StreamingDataLoader to be able to fully utilise the GPU - but it doesn't play nicely with map because we don't set the right env variables and things for it to work in a distributed setting.
### Motivation
This would let us run distributed embedding of much larger data sets like LAION
### Pitch
Allow providing a StreamingDataLoader to the map function, and then set correct envs etc. so that we still visit each sample just once.
Contributor guide
Research direction
Start at the map entry point and StreamingDataLoader implementation, then trace how distributed environment variables are handled. Define the integration needed for passing a StreamingDataLoader to map, with completion demonstrated by distributed embedding work that visits each sample exactly once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100