Cannot pickle objects when using unix socket
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 13d 8h
- Merged PRs (30d)
- 2
Description
One of my use-cases is loading a list of containers in one process and then pushing them to be processed in another process, using a standard python queue.
While this method works well with the tcp:// socket, when using the unix:// socket the pickler will throw the following exception:
queue.put((QUEUE_OBJ, obj), block=block)
File "<string>", line 2, in put
File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 756, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'UnixAdapter.__init__.<locals>.<lambda>'
On one hand I could just push the IDs of the objects downstream, but I need some of the attributes belonging to the Containers as well, and it makes for quite an inconvenient way of sharing them.
I am not sure if that is the expected behaviour; but the TCP protocol works well enough and I can't see a reason why UNIX should not.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with a standard multiprocessing queue and the unix:// transport, then inspect UnixAdapter.init and the multiprocessing pickling path shown in the traceback. Done means objects containing container attributes can be sent through a unix socket without the local-lambda pickling error, while the existing tcp:// behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100