Lightning-AI / Lightning-AI/litData

Add support for TiffStreamingRawDataset

Open
#687 1 comment 0 reactions 1 assignee View on GitHub

@GSNCodes is already working on this.

Since Nov 23, 2025.

enhancement help wanted
Dominant language
Python
Stars
614
Forks
106
Avg merge
15h 8m
Merged PRs (30d)
22

Description

🚀 Feature

Notes from @tchaton

We could add https://developmentseed.org/async-tiff/latest to the StreamingRawDataset

from litdata import StreamingRawDataset
from litdata.raw.types import TIFF
import torch

class TiffStreamingRawDataset(StreamingRawDataset):

    def setup(self, urls):
        return [TIFF(url, tile=(512, 512, 3), ....]

    def __getitem__(self, decoded_bytes: bytes):
        return torch.frombuffer(decoded_bytes, torch.uint8)

example: https://github.com/microsoft/pytorch-cloud-geotiff-optimization/blob/5fb6d1294163beff822441829dcd63a3791b7808/optimized_cog_streaming/datamodules.py#L89 and https://github.com/microsoft/pytorch-cloud-geotiff-optimization/blob/5fb6d1294163beff822441829dcd63a3791b7808/optimized_cog_streaming/datasets.py#L42

Motivation
Pitch
Alternatives
Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.