Lightning-AI / Lightning-AI/litData

Support `StreamingDataLoader` passed to `map`

Open
#13 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
614
Forks
106
Avg merge
15h 8m
Merged PRs (30d)
22

Description

## 🚀 Feature

It would be great to be able to pass a StreamingDataLoader to map. When experimenting with CLIP embeddings I've found that I needed to use StreamingDataLoader to be able to fully utilise the GPU - but it doesn't play nicely with map because we don't set the right env variables and things for it to work in a distributed setting.

### Motivation

This would let us run distributed embedding of much larger data sets like LAION

### Pitch

Allow providing a StreamingDataLoader to the map function, and then set correct envs etc. so that we still visit each sample just once.

Contributor guide

Open the contributing guide

Research direction

Start at the map entry point and StreamingDataLoader implementation, then trace how distributed environment variables are handled. Define the integration needed for passing a StreamingDataLoader to map, with completion demonstrated by distributed embedding work that visits each sample exactly once.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, distributed-systems, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.