docker / docker/docker-py

Performance of `consume_socket_output` is n² where it could be just `n`

Open
#3,349 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.2k
Forks
1.7k
Avg merge
13d 8h
Merged PRs (30d)
2

Description

I noticed that the function is repeatedly concatenating bytes:

def consume_socket_output(frames, demux=False):
    ...
    # If the streams are demultiplexed, the generator yields tuples
    # (stdout, stderr)
    out = [None, None]
    for frame in frames:
        # It is guaranteed that for each frame, one and only one stream
        # is not None.
        assert frame != (None, None)
        if frame[0] is not None:
            if out[0] is None:
                out[0] = frame[0]
            else:
                out[0] += frame[0]
        else:
            if out[1] is None:
                out[1] = frame[1]
            else:
                out[1] += frame[1]
    return tuple(out)

This makes it extremely slow for long inputs. I noticed since I used dockerpy to get about 1GB of process output, which took hours and blocked 100% CPU for the entire time.

The fix for it should be pretty simple, just collect all the frames and use b''.join(frames) instead:

# If the streams are demultiplexed, the generator yields tuples
# (stdout, stderr)
stdout = []
stderr = []
for stdout_frame, stderr_frame in frames:
    # It is guaranteed that for each frame, one and only one stream
    # is not None.
    if stdout_frame:
        stdout.append(stdout_frame)
    else:
        stderr.append(stderr_frame)
stdout = b''.join(stdout) if len(stdout) > 0 else None
stderr = b''.join(stderr) if len(stderr) > 0 else None
return stdout, stderr

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in docker/utils/socket.py at consume_socket_output, around the linked lines, and inspect how demultiplexed frames are accumulated. Verify the change with long stdout and stderr inputs, ensuring the returned streams preserve frame order and empty streams remain None.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.