cloudpipe / cloudpipe/cloudpickle

Can not pickle `torch.utils.data.DataLoader`

Open
#587 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
195
Avg merge
1d 10h
Merged PRs (30d)
1

Description

Hi team,

I'm working on some distributed system for AI, and I'd like to pickle an object to transfer it. But it seems I can not pickle DataLoader, is there any suggestions or best practice for that?

```python
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/flamepy/client.py", line 325, in create_session
object_ref = put_object(session_id, attrs.common_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/flamepy/cache.py", line 74, in put_object
data = cloudpickle.dumps(obj, protocol=cloudpickle.DEFAULT_PROTOCOL)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1544, in dumps
cp.dump(obj)
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1313, in dump
return super().dump(obj)
^^^^^^^^^^^^^^^^^
File "/opt/examples/ps/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 761, in __getstate__
raise NotImplementedError("{} cannot be pickled", self.__class__.__name__)
NotImplementedError: ('{} cannot be pickled', '_SingleProcessDataLoaderIter')
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.