Lightning-AI / Lightning-AI/litData
index_path is ignored in StreamingDataset when input_dir is a local directory
- Dominant language
- Python
- Stars
- 614
- Forks
- 106
- Avg merge
- 15h 8m
- Merged PRs (30d)
- 22
Description
## 🐛 Bug
It is not possible to use a custom `index_path` with `StreamingDataset` when `input_dir` is a local path. This appears to be caused by a check in `subsample_streaming_dataset` that determines whether `input_dir` is a URL:
https://github.com/Lightning-AI/litData/blob/36431bde711c17b92d6f67f9ea92fa5bb47ba739/src/litdata/utilities/dataset_utilities.py#L83
### To Reproduce
As a simple example, download a small dataset from Hugging Face as Parquet files and index it with `index_parquet_dataset`:
```python
from huggingface_hub import snapshot_download
import litdata as ld
def main():
snapshot_download(repo_id="roneneldan/TinyStories", local_dir="tiny_stories_data", repo_type="dataset")
ld.index_parquet_dataset("tiny_stories_data/data", cache_dir="my-custom-cache")
if __name__ == "__main__":
main()
```
Then create a `StreamingDataset` with:
```python
dataset = StreamingDataset("tiny_stories_data/data", shuffle=True, index_path="my-custom-cache/index.json", item_loader=ParquetLoader())
```
This raises:
```bash
ValueError: The provided dataset `tiny_stories_data/data` doesn't contain any index.json file.
HINT: Did you successfully optimize a dataset to the provided `input_dir`?
```
### Expected behavior
The `index_path` docstring states:
```
Path to index.json for the Parquet dataset. If index_path is a directory, the function will look for index.json within it. If index_path is a full file path, it will use that directly.
```
Therefore, to my understanding, `StreamingDataset` should accept an `index_path` that points to an `index.json` located outside `input_dir` (for example, in a separate folder) and use that index regardless of whether `input_dir` is a local path or a URL.
### Additional context
The issue can be worked around by prefixing the path with `local:`, but the documentation does not make it clear whether that prefix should be required. Thank you in advance!
Contributor guide
Research direction
Start in src/litdata/utilities/dataset_utilities.py around the input_dir URL check at line 83, then run the supplied local Parquet reproduction using index_parquet_dataset and StreamingDataset. Done means a local input_dir accepts an index_path outside it and uses the specified index.json without requiring the local: prefix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100