Lightning-AI/litData

View on GitHub

Speed up model training by fixing data loading.

Stars: 614
Forks: 106
Open beginner issues: 0
Indexed issues: 41
Avg merge: 15h 8m
Merged PRs (30d): 22

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 11, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: help wanted good first issue

Beginner friendly Everything

Open All

41 open issues indexed

Checkpoint uploads truncate output prefix for stable checkpoint filenames Open

bug help wanted

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

Lightning-AI/litData#888 · Opened Aug 18, 2026
Concurrency Budget is Static & Estimated From Hardcoded Magic Constants Open

bug help wanted

Difficulty 5/5 Over a week Newbie friendliness 35/100

Lightning-AI/litData#870 · Opened Aug 10, 2026
parquet tests fails inconsistently in CI Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/litData#813 · 1 comment · Opened Apr 29, 2026
index_path is ignored in StreamingDataset when input_dir is a local directory Open

bug help wanted

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

Lightning-AI/litData#800 · 1 comment · 1 reaction · Opened Mar 19, 2026
Shuffle buffer for controlling sample correlation Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

Lightning-AI/litData#797 · 2 comments · Opened Mar 11, 2026
Add Global max_cache_size to CombinedStreamingDataset to Enforce a Total Cache Budget Across Child Datasets Open

enhancement

Lightning-AI/litData#790 · 1 comment · 1 reaction · 1 assignee · Opened Feb 17, 2026
Interrupt and resume merge_datasets Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 45/100

Lightning-AI/litData#784 · 2 comments · 1 reaction · Opened Jan 14, 2026
`StreamingDataLoader` state is not loaded from checkpoint when resuming training Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 43/100

Lightning-AI/litData#775 · 1 comment · 1 reaction · Opened Dec 19, 2025
TokensLoader segmentation fault Open

bug help wanted waiting on author

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Lightning-AI/litData#756 · 1 comment · Opened Nov 20, 2025
TokensLoader with parallel meta field Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 45/100

Lightning-AI/litData#753 · Opened Nov 15, 2025
Supporting samplers for creating balanced batches Open

question waiting on author

Difficulty 5/5 Over a week Newbie friendliness 35/100

Lightning-AI/litData#717 · 1 comment · Opened Sep 22, 2025
Add support for Teamspaces EFS and Filestore folders in resolver Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 45/100

Lightning-AI/litData#698 · Opened Aug 23, 2025
Add support for raw `Video` types for sliced video loading Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

Lightning-AI/litData#688 · 1 reaction · Opened Aug 10, 2025
Add support for TiffStreamingRawDataset Open

enhancement help wanted

Lightning-AI/litData#687 · 1 comment · 1 assignee · Opened Aug 10, 2025
Switch raw dataset indexing from `fsspec` to `obstore` Open

enhancement help wanted

Difficulty 5/5 Over a week Newbie friendliness 35/100

Lightning-AI/litData#686 · 1 comment · Opened Aug 10, 2025
Add support for easy logging setup with sensible defaults Open

enhancement good first issue help wanted

Lightning-AI/litData#685 · 2 comments · 2 reactions · 1 assignee · Opened Aug 10, 2025
Try `obstore` for LitData Chunks Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/litData#663 · 2 comments · Opened Jul 27, 2025
Support Cycling a CombinedStreamingDataset Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

Lightning-AI/litData#658 · 3 comments · 1 reaction · Opened Jul 22, 2025
Support for subclasses of `StreamingDataset` with different length Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

Lightning-AI/litData#642 · 1 comment · 3 reactions · Opened Jun 30, 2025
Explore Optimizing and Running Tests in Parallel for Faster CI in `litdata` Open

ci / tests help wanted

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/litData#612 · 3 comments · Opened Jun 4, 2025
Add example for hugging face dataset optimization and benchmarks reports Open

documentation

Lightning-AI/litData#558 · 1 reaction · 1 assignee · Opened Apr 17, 2025
Cycle option for StreamingDataLoader Open

enhancement waiting on author

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Lightning-AI/litData#524 · 8 comments · Opened Mar 24, 2025
Local cache dir not fully clearing in DDP multi-node training. Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 25/100

Lightning-AI/litData#512 · 12 comments · 1 reaction · Opened Mar 12, 2025
Add batch sampler for StreamDataloader to enable flexiable training strategy. Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Lightning-AI/litData#509 · 2 comments · 2 reactions · Opened Mar 11, 2025
CI error: `All chunks should've been deleted` keeps coming back Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 25/100

Lightning-AI/litData#437 · 2 comments · 1 reaction · Opened Dec 20, 2024
Restart training with new data, mid-epoch Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

Lightning-AI/litData#436 · 1 comment · 3 reactions · Opened Dec 17, 2024
Clear Examples of use with different dataset types and code changes. Open

documentation enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/litData#409 · 3 comments · Opened Nov 4, 2024
incorrect dataloader length when `drop_last=False` Open

bug help wanted

Difficulty 3/5 1-2 days Newbie friendliness 38/100

Lightning-AI/litData#402 · 2 comments · 2 reactions · Opened Oct 28, 2024
Improve CombinedStreamingDataset to handle multiple subdatasets efficiently Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

Lightning-AI/litData#386 · 2 comments · 2 reactions · Opened Oct 2, 2024
The config isn't consistent between chunks Open

bug help wanted waiting on author

Difficulty 4/5 3-5 days Newbie friendliness 32/100

Lightning-AI/litData#370 · 11 comments · Opened Sep 17, 2024
Investigate keeping the content of the downloaded chunks in RAM instead of writing it to file. Open

enhancement help wanted

Difficulty 5/5 Over a week Newbie friendliness 20/100

Lightning-AI/litData#291 · 5 comments · 2 reactions · Opened Aug 1, 2024
Add training mode compression for zstd Open

enhancement help wanted

Difficulty 5/5 Over a week Newbie friendliness 25/100

Lightning-AI/litData#283 · 3 comments · Opened Jul 31, 2024
Add support for sample windowing Open

enhancement help wanted

Difficulty 5/5 Over a week Newbie friendliness 35/100

Lightning-AI/litData#282 · 6 comments · Opened Jul 31, 2024
RuntimeError: Can't start new thread Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 30/100

Lightning-AI/litData#280 · 12 comments · Opened Jul 31, 2024
If data samples contain a Python list with a variable number of elements, type inference will fail Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/litData#260 · 11 comments · 1 reaction · Opened Jul 23, 2024
Integration with DAG framework Prefect Open

enhancement help wanted

Lightning-AI/litData#226 · 1 comment · 1 assignee · Opened Jul 12, 2024
Add support for the reduce operator Open

enhancement help wanted won't fix

Difficulty 5/5 Over a week Newbie friendliness 25/100

Lightning-AI/litData#225 · 4 comments · Opened Jul 12, 2024
Using a streaming dataloader with an unbalanced dataset yields unexpected batch sizes. Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/litData#199 · 10 comments · 2 reactions · Opened Jun 29, 2024
Pytorch lighting Fabric + lit data + DDP hangs when finishing epoch Open

bug help wanted

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/litData#129 · 6 comments · Opened May 13, 2024
Allow a StreamingDataset to wrap around when running in a CombinedStreamingDataset Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/litData#74 · 5 comments · Opened Mar 14, 2024
Support `StreamingDataLoader` passed to `map` Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

Lightning-AI/litData#13 · 4 comments · Opened Feb 23, 2024

Lightning-AI/litData

Get new issues in your inbox