mosaicml

Data loss: cache_limit can evict canonical local-only shards when remote=None Open

Difficulty 3/5 1-2 days Newbie friendliness 72/100

mosaicml/streaming#988 · Opened Aug 17, 2026

Are there any supported patterns for on-the-fly caching? Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#981 · Opened May 1, 2026

Not compatible with HF transformers >=5.0.0 Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

mosaicml/streaming#979 · 3 comments · 2 reactions · Opened Mar 12, 2026

Does MDSWriter release the GIL? Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mosaicml/streaming#978 · Opened Mar 1, 2026

Support one to many mapping? Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#974 · Opened Feb 11, 2026

Support for different OCI object store authentication mechanisms Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 42/100

mosaicml/streaming#973 · Opened Feb 6, 2026

dataframe_to_mds fails on large datasets due to Worker Disk Exhaustion Open

bug

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#970 · 1 comment · Opened Feb 2, 2026

Windows absolute paths with drive letters incorrectly parsed as cloud provider schemes Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

mosaicml/streaming#960 · 1 comment · Opened Nov 4, 2025

merge_index drops full directory path in merged index.json, causing broken shard references and 404s Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 68/100

mosaicml/streaming#959 · 2 comments · Opened Nov 3, 2025

Performance imporve: mmap and file handler cache Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#958 · 2 comments · Opened Oct 31, 2025

Load only a subset of columns Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#926 · 1 reaction · Opened Jun 3, 2025

Feature request: support for split/Subset-like behavior and shared local cache in StreamingDataset Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#916 · Opened May 15, 2025

How to sample nearby items while keeping performance? Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#905 · Opened Apr 22, 2025

Allow configuring max_pool_connections on boto3 s3 client Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

mosaicml/streaming#902 · Opened Apr 19, 2025

Race Condition in base.shared.get_shm_prefix (possible general SharedMemory issue) Open

bug

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#901 · 8 comments · Opened Apr 14, 2025

Clean up garbage files Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#893 · 2 comments · Opened Apr 2, 2025

StreamingDataset gives FileExistsError when called with multiprocessing Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mosaicml/streaming#884 · 1 reaction · Opened Mar 10, 2025

Unable to get mid-epoch resumption working Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#880 · Opened Feb 25, 2025

Streaming Dataset Simulator stops working beyond 10,000 batches Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#877 · 1 comment · Opened Feb 12, 2025

NCCL Timeouts from Adding More Datasets + Proportion Sampling Open

bug

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#876 · 6 comments · Opened Feb 11, 2025

[Bug] Incorrect local file path in HfUploader Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 48/100

mosaicml/streaming#875 · Opened Feb 9, 2025

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/mds_data/train/index.json' Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mosaicml/streaming#873 · Opened Feb 7, 2025

exist_ok mds_kwargs not passable to dataframe_to_mds() Open

Difficulty 2/5 1-3 hours Newbie friendliness 52/100

mosaicml/streaming#871 · 1 reaction · Opened Jan 31, 2025

dataframe_to_mds fails with non-nullable ArrayType Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 52/100

mosaicml/streaming#870 · 3 comments · Opened Jan 30, 2025

Facing issue with resuming training for saved dataset state (>1 epoch) Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#869 · 3 comments · Opened Jan 28, 2025

Facing issue with improper file renaming while multi-node training in ROCm Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#868 · 4 comments · Opened Jan 27, 2025

how about read speed when training, just save on local. compare to use pytorch dataloader Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mosaicml/streaming#864 · 1 comment · Opened Jan 20, 2025

Error in Streaming Dataset Decompression in Distributed Setting Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#863 · 2 comments · 1 reaction · Opened Jan 15, 2025

Expose StreamingDataset world (or world_size and rank) as argument Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#854 · 1 reaction · Opened Dec 19, 2024

Will cache eviction logic take previously-existing shards into account? Open

Difficulty 3/5 1-2 days Newbie friendliness 30/100

mosaicml/streaming#844 · 3 comments · Opened Dec 5, 2024

Support for Lance Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 20/100

mosaicml/streaming#832 · 5 comments · Opened Nov 20, 2024

Pipeline Parallelism (Supported? How to?) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#827 · 6 comments · Opened Nov 14, 2024

Cannot Load MDS Dataset Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mosaicml/streaming#824 · 43 comments · Opened Nov 11, 2024

UnicodeDecodeError: ... Efficient way to debug the dataset with streaming? Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 20/100

mosaicml/streaming#820 · 4 comments · Opened Nov 1, 2024

write to S3 is very slow Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#812 · 3 comments · Opened Oct 25, 2024

Choose JPEG compression level Open

enhancement

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

mosaicml/streaming#811 · 1 comment · Opened Oct 24, 2024

MosaicML-Streaming on Databricks Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#801 · 9 comments · Opened Oct 10, 2024

Support for on-the-fly filtering Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#800 · 1 comment · 8 reactions · Opened Oct 9, 2024

Make `epoch_sample_ids` cachable Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#792 · 2 comments · Opened Sep 28, 2024

Dataset does not work after stopping training Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 30/100

mosaicml/streaming#781 · 1 comment · Opened Sep 15, 2024

Sparse Numpy Arrays Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#780 · Opened Sep 10, 2024

JointWriter: Allow shard file appending Open

bug

Difficulty 5/5 Over a week Newbie friendliness 35/100

mosaicml/streaming#775 · 2 comments · Opened Sep 5, 2024

MemoryError: Unable to allocate Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 28/100

mosaicml/streaming#771 · 1 comment · Opened Sep 2, 2024

File exists: '/000000_epoch_shape' when using the ddp strategy from pytorch lightning Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mosaicml/streaming#767 · 23 comments · 1 reaction · Opened Aug 25, 2024

Estimate total shards at the beginning of data conversion Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mosaicml/streaming#742 · 1 comment · Opened Aug 3, 2024

huge temp files while uploading data using MDS writer Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 30/100

mosaicml/streaming#734 · 2 comments · Opened Jul 24, 2024

Download optimal for device_per_stream batching method. Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mosaicml/streaming#726 · 4 comments · Opened Jul 16, 2024

Replication changes sample order Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mosaicml/streaming#725 · 3 comments · Opened Jul 15, 2024

'File exists: "/00000_locals"' when integrated with deepspeed training scripts Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 28/100

mosaicml/streaming#717 · 4 comments · 3 reactions · Opened Jul 8, 2024

All processes allocate memory on rank 0 during StreamingDataset initialization in a distributed setting Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mosaicml/streaming#716 · 4 comments · Opened Jul 2, 2024

Get new issues in your inbox