Performance of `consume_socket_output` is n² where it could be just `n`
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 13d 8h
- Merged PRs (30d)
- 2
Description
I noticed that the function is repeatedly concatenating bytes:
def consume_socket_output(frames, demux=False):
...
# If the streams are demultiplexed, the generator yields tuples
# (stdout, stderr)
out = [None, None]
for frame in frames:
# It is guaranteed that for each frame, one and only one stream
# is not None.
assert frame != (None, None)
if frame[0] is not None:
if out[0] is None:
out[0] = frame[0]
else:
out[0] += frame[0]
else:
if out[1] is None:
out[1] = frame[1]
else:
out[1] += frame[1]
return tuple(out)
This makes it extremely slow for long inputs. I noticed since I used dockerpy to get about 1GB of process output, which took hours and blocked 100% CPU for the entire time.
The fix for it should be pretty simple, just collect all the frames and use b''.join(frames) instead:
# If the streams are demultiplexed, the generator yields tuples
# (stdout, stderr)
stdout = []
stderr = []
for stdout_frame, stderr_frame in frames:
# It is guaranteed that for each frame, one and only one stream
# is not None.
if stdout_frame:
stdout.append(stdout_frame)
else:
stderr.append(stderr_frame)
stdout = b''.join(stdout) if len(stdout) > 0 else None
stderr = b''.join(stderr) if len(stderr) > 0 else None
return stdout, stderr
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in docker/utils/socket.py at consume_socket_output, around the linked lines, and inspect how demultiplexed frames are accumulated. Verify the change with long stdout and stderr inputs, ensuring the returned streams preserve frame order and empty streams remain None.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100