Lightning-AI / Lightning-AI/litData
Add support for TiffStreamingRawDataset
Open
@GSNCodes is already working on this.
Since Nov 23, 2025.
enhancement
help wanted
- Dominant language
- Python
- Stars
- 614
- Forks
- 106
- Avg merge
- 15h 8m
- Merged PRs (30d)
- 22
Description
🚀 Feature
Notes from @tchaton
We could add https://developmentseed.org/async-tiff/latest to the StreamingRawDataset
from litdata import StreamingRawDataset
from litdata.raw.types import TIFF
import torch
class TiffStreamingRawDataset(StreamingRawDataset):
def setup(self, urls):
return [TIFF(url, tile=(512, 512, 3), ....]
def __getitem__(self, decoded_bytes: bytes):
return torch.frombuffer(decoded_bytes, torch.uint8)
example: https://github.com/microsoft/pytorch-cloud-geotiff-optimization/blob/5fb6d1294163beff822441829dcd63a3791b7808/optimized_cog_streaming/datamodules.py#L89 and https://github.com/microsoft/pytorch-cloud-geotiff-optimization/blob/5fb6d1294163beff822441829dcd63a3791b7808/optimized_cog_streaming/datasets.py#L42
Motivation
Pitch
Alternatives
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.