Lightning-AI / Lightning-AI/litData

index_path is ignored in StreamingDataset when input_dir is a local directory

Open Beginner friendly
#800 1 comment 1 reaction 0 assignees View on GitHub
bug help wanted
Dominant language
Python
Stars
614
Forks
106
Avg merge
15h 8m
Merged PRs (30d)
22

Description

## 🐛 Bug

It is not possible to use a custom `index_path` with `StreamingDataset` when `input_dir` is a local path. This appears to be caused by a check in `subsample_streaming_dataset` that determines whether `input_dir` is a URL:
https://github.com/Lightning-AI/litData/blob/36431bde711c17b92d6f67f9ea92fa5bb47ba739/src/litdata/utilities/dataset_utilities.py#L83

### To Reproduce

As a simple example, download a small dataset from Hugging Face as Parquet files and index it with `index_parquet_dataset`:

```python
from huggingface_hub import snapshot_download
import litdata as ld

def main():
snapshot_download(repo_id="roneneldan/TinyStories", local_dir="tiny_stories_data", repo_type="dataset")
ld.index_parquet_dataset("tiny_stories_data/data", cache_dir="my-custom-cache")

if __name__ == "__main__":
main()
```

Then create a `StreamingDataset` with:

```python
dataset = StreamingDataset("tiny_stories_data/data", shuffle=True, index_path="my-custom-cache/index.json", item_loader=ParquetLoader())
```

This raises:

```bash
ValueError: The provided dataset `tiny_stories_data/data` doesn't contain any index.json file.
HINT: Did you successfully optimize a dataset to the provided `input_dir`?
```

### Expected behavior

The `index_path` docstring states:
```
Path to index.json for the Parquet dataset. If index_path is a directory, the function will look for index.json within it. If index_path is a full file path, it will use that directly.
```

Therefore, to my understanding, `StreamingDataset` should accept an `index_path` that points to an `index.json` located outside `input_dir` (for example, in a separate folder) and use that index regardless of whether `input_dir` is a local path or a URL.

### Additional context

The issue can be worked around by prefixing the path with `local:`, but the documentation does not make it clear whether that prefix should be required. Thank you in advance!

Contributor guide

Open the contributing guide

Research direction

Start in src/litdata/utilities/dataset_utilities.py around the input_dir URL check at line 83, then run the supplied local Parquet reproduction using index_parquet_dataset and StreamingDataset. Done means a local input_dir accepts an index_path outside it and uses the specified index.json without requiring the local: prefix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.