RFC: Deprecation of MapStyle Dataset

tl;dr RFC: Do you need to access dataset by index?

Currently torchaudio provides map-style datasets, which you can get a sample data with indexing.

dataset = torchaudio.datasets.VCTK("foo")
sample = dataset[3]

There is an ongoing efforts to improve data loading mechanism on PyTorch (see https://github.com/pytorch/pytorch/issues/41292), and the proposal along with it is DataPipe which will be a new component to build dataset.

The following is an illustration code snippet, taken from https://github.com/pytorch/pytorch/issues/49440

list_files_dp = datapipes.map.ListFiles(root = '.')
load_bins_dp = datapipes.map.LoadFiles(list_files_dp)
decode_images_dp = datapipes.map.DecodeImages(load_bins_dp)
transform_dp = datapipes.map.TransformImages(decode_images_dp)
shuffle_dp = datapipes.map.Shuffle(transform_dp)
sampler_dp = datapipes.iter.Sampler(shuffle_dp) 

Writing dataset in DataPipe will allow us to make the primitives reusable and allow the future DataLoader better utilize the subprocesses (sharding). However to take the advantage of efficient data loading the future DataLoader will provide, it is suggested to implement Dataset as iterable, instead of map-style.

We would like to learn if there is a strong, compelling use case for map-style dataset. (Something only map-style dataset can achieve well) Note that shuffling will be supported in the DataPipe-based dataset implementation.

At the moment, we do not have a plan to change the existing datasets, and if we ever do, we will communicate with users through issues and release notes, but we would like to hear at this early stage if there is some use cases that we are not aware of.

Please leave a comment if you have an idea or question. Thanks!

This is an RFC and names no source files, tests, or implementation entry points. Read the linked PyTorch issues on DataLoader and DataPipe, then evaluate whether any compelling use case requires map-style datasets. The discussion is complete when a concrete use case or question is documented in the issue comments.