whether to support streaming dataset？
Open
type:feature
- Dominant language
- Python
- Stars
- 779
- Forks
- 86
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 6
Description
Whether grain provides iter dataset, similar to torch.utils.data.IterableDataset, because when the total amount of original index files is large, such as 4T, they are difficult to load directly into memory.
Contributor guide
Assessment
This issue has not been assessed yet.