docker / docker/docker-py

Cannot pickle objects when using unix socket

Open
#1,809 8 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.2k
Forks
1.7k
Avg merge
13d 8h
Merged PRs (30d)
2

Description

One of my use-cases is loading a list of containers in one process and then pushing them to be processed in another process, using a standard python queue.

While this method works well with the tcp:// socket, when using the unix:// socket the pickler will throw the following exception:

    queue.put((QUEUE_OBJ, obj), block=block)
  File "<string>", line 2, in put
  File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 756, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/local/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'UnixAdapter.__init__.<locals>.<lambda>'

On one hand I could just push the IDs of the objects downstream, but I need some of the attributes belonging to the Containers as well, and it makes for quite an inconvenient way of sharing them.

I am not sure if that is the expected behaviour; but the TCP protocol works well enough and I can't see a reason why UNIX should not.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with a standard multiprocessing queue and the unix:// transport, then inspect UnixAdapter.init and the multiprocessing pickling path shown in the traceback. Done means objects containing container attributes can be sent through a unix socket without the local-lambda pickling error, while the existing tcp:// behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.