datapipe serialization support / cloudpickle / parallel support

I've been looking at how we might go about supporting torchdata within TorchX and with components. I was wondering what the serialization options were for transforms and what that might look like.

There's a couple of common patterns that would be nice to support:

• general data transforms (with potentially distributed preprocessing via torch elastic/ddp)
• data splitting into train/validation sets
• summary statistic computation

For the general transforms and handling arbitrary user data we were wondering how we might go about serializing the data pipes and transforms for use in a pipeline with TorchX.

There's a couple of options here:

1. add serialization support to the transforms so you can serialize them (lambdas?)
2. generate a .py file from a provided user function
3. pickle the transform using something like cloudpickle/torch.package and load it in a trainer app
4. ask the user to write a .py file that uses the datapipes as the transform and create a TorchX component (what we currently have)

Has there been any thought about how to support this well? Is there extra work that should be done here to make this better?

Are DataPipes guaranteed to be pickle safe and is there anything that needs to be done to support that?

I was also wondering if there's multiprocessing based datapipes and how that works since this seems comparable. I did see https://github.com/pytorch/pytorch/blob/master/torch/utils/data/distributed.py but didn't see any examples on how to use that to achieve a traditional PyTorch dataloader style workers.

P.S. should this be on the pytorch discussion forums instead? it's half feature request half questions so wasn't sure where best to put it

cc @kiukchung

Start with the referenced torch/utils/data/distributed.py and compare the proposed cloudpickle, torch.package, generated-file, and user-authored-file approaches for TorchX and DataPipes. No concrete tests, files, or acceptance criteria are named; define the desired serialization and parallel-processing behavior before implementation.