cloudpipe / cloudpipe/cloudpickle
Can not pickle `torch.utils.data.DataLoader`
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 195
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 1
Description
Hi team,
I'm working on some distributed system for AI, and I'd like to pickle an object to transfer it. But it seems I can not pickle DataLoader, is there any suggestions or best practice for that?
```python
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/flamepy/client.py", line 325, in create_session
object_ref = put_object(session_id, attrs.common_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/flamepy/cache.py", line 74, in put_object
data = cloudpickle.dumps(obj, protocol=cloudpickle.DEFAULT_PROTOCOL)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1544, in dumps
cp.dump(obj)
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1313, in dump
return super().dump(obj)
^^^^^^^^^^^^^^^^^
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 761, in __getstate__
raise NotImplementedError("{} cannot be pickled", self.__class__.__name__)
NotImplementedError: ('{} cannot be pickled', '_SingleProcessDataLoaderIter')
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.