meta-pytorch / meta-pytorch/data
Add backend switch for `IterDataPipe.__getstate__`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
I see. I can confirm that we will rely on this
DILL_AVAILABLEfor any place in TorchData project to determine ifdillis available or not.
The problem is that we will need to patch every single module where this is imported. You cannot patch the place where it is defined, but rather where it is used. If you look above, we are not patching ._utils.serialization but rather .datapipes.datapipe, because this is where the flag is used. If we now need use this flag in multiple modules, we need to patch all of them. This is very brittle.
It's doable, but I am not sure if we want to do so because the goal of automatically using
dillis to reduce the work users need to figure out if the DataPipe is serializable with lambda function.
Not sure I understand. If we just keep the same detection as we have now, users that don't care should not see any difference. If dill is available, it will be picked up and otherwise pickle will be used. But it would give users the option to enforce a particular backend if they need to. Without this option, the environment you use has an effect on the functionality and there is no way change that. I don't think this is good design.
Even if you don't do it for the users, think about how you want to test pickle vs dill yourself. Right now the only option is to have two separate workflows one with dill installed and one without.
Originally posted by @pmeier in https://github.com/pytorch/vision/pull/5711#discussion_r840280966
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing IterDataPipe.getstate and the DILL_AVAILABLE usage described in the discussion, then inspect the existing serialization path. Done should mean users can explicitly choose pickle or dill rather than being limited by environment detection, with tests covering both backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100