Unexpected EOF, when streaming output via exec_run()
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 13d 8h
- Merged PRs (30d)
- 2
Description
Ubuntu: 20.04.1 LTS
Docker version 20.10.21, build 20.10.21-0ubuntu1~20.04.2
Docker-Py: 6.0.1
I'm running into an issue where I hit this line of code, before a command invoked by exec_run() is fully completed serving all the output through its generator: https://github.com/docker/docker-py/blob/84414e343e526cf93f285284dd2c2c40f703e4a9/docker/utils/socket.py#L71
I'm executing something similar to the following (the actual command differs, but is effectively something like this):
bash -c "{ long_running_command.sh; echo -n $? }"
The setup is such that using docker-py 6.0.1, I'm invoking this command via exec_run() where stream=True, to execute a composition of ; separated bash commands, finishing off with echo-ing a special string to signal the completion of the overall command. In the high 90 percentile of cases, this works perfectly well. However, sporadically and what appears randomly, we don't get to the point where we print that special string, since we haven't received all the output that was printed by the previous command (aka: long_running_command.sh).
From my end, the symptoms are such that the generator is done serving us additional output, and my code resumes execution, despite the fact that not all the output from long_running_command.sh has been received.
This happens across several test machines, multiple versions of docker, and multiple versions of docker-py.
Upon further investigation, it seems that the generator that returns the output of the streamed exec_run() invocation returns before all the output has been served, and upon further investigation, it appears that the above referenced line is being hit in such cases.
If a retry of the exact command is performed, it succeeds (in all instances that this has been attempted, anyways). This seems to me like a spontaneous Unix socket disconnect, when docker-py is trying to get the next header.
https://github.com/docker/docker-py/blob/84414e343e526cf93f285284dd2c2c40f703e4a9/docker/utils/socket.py#L84
Wondering if anyone is aware of this, has noticed this, or what data I can collect to be able to further diagnose.
Thank you in advance.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docker/utils/socket.py at the referenced lines 71 and 84, then reproduce exec_run() with stream=True and the long-running command plus completion marker. Compare the generator's end with the received output and investigate the reported Unix socket disconnect. Done means the premature completion is reproduced and its expected handling is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- api, backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100