not all prototype transforms are serializable
Open
@pmeier is already working on this.
Since Oct 10, 2022.
bug
module: transforms
prototype
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 7.3k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 13
Description
This is important for two reasons:
- If we stick to
torchdatafor datasets v2, every object on the datapipe graph has to be serializable for multiprocessing. IIRC, this is only a requirement if forking from the main process is not possible, but that is the case on Windows. - Our classification references have the ability to cache the dataset which includes the transformations:
So far we don't have a unified test framework for our prototype transforms and thus it is not easy to add tests for all transforms. Note that we don't need tests for the functional kernels and dispatchers, since they are already tested for JIT scriptability and that protocol is a lot stricter than pickling.
cc @vfdev-5 @datumbox @bjuncek
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.