missing output using exec_run

Hi

I use this library to test some handler that manage containers. I'm getting an issue that's make my integration test blinks with false negative.

While using exec_run on a running container, sometimes output is an empty string where I'm expecting
some results.

(Pdb) containers.get(container).exec_run('echo "hello"').output.decode('utf-8')
'hello\n'
(Pdb) containers.get(container).exec_run('echo "hello"').output.decode('utf-8')
'hello\n'
(Pdb) containers.get(container).exec_run('echo "hello"').output.decode('utf-8')
'hello\n'
(Pdb) containers.get(container).exec_run('echo "hello"').output.decode('utf-8')
''
(Pdb) containers.get(container).exec_run('echo "hello"').output.decode('utf-8')
'hello\n'

I'm not sure if the matter is related to this library or to the docker daemon. My understanding is that the command is executed before the stream is reading outputs.

A workaround I use is to sleep a bit to make sure getting logs in output likes:

(Pdb) containers.exec_run('bash -c "sleep 0.1; echo hello"').output
'hello\n'

My environment is a bit weird, python code is launched on my Debian laptop using python 3.5.3, docker v 3.3.0 using Docker version 18.03.1-ce, build 9ee9f40

The daemon is running on Container Linux by CoreOS 1745.4.0 (Rhyolite) VM launched by virsh on my laptop using by chance the same
Docker version 18.03.1-ce, build 9ee9f40

I'm binding the socket through ssh like this ssh -nNT -L /tmp/docker_core1.sock:/var/run/docker.sock -i ../salt/srv/base/ssh/core_id_rsa core@192.168.122.95

And instantiate Docker client like this:

containers = docker.DockerClient(base_url="unix:///tmp/docker_core1.sock").containers

Unfortunately I haven't be able to reproduce using the local docker daemon on my laptop only using the coreos daemon through the ssh tunnel:

# python3 -m venv exec_run_ningx
# source exec_run_ningx/bin/activate
(exec_run_ningx) # pip install docker
Collecting docker
  Downloading https://files.pythonhosted.org/packages/31/01/41f727a549abd788eb91d0df4c801cb4c1f5227eee04a2621e13d7ed3e04/docker-3.3.0-py2.py3-none-any.whl (124kB)
    100% |████████████████████████████████| 133kB 1.1MB/s 
Collecting requests!=2.18.0,>=2.14.2 (from docker)
  Downloading https://files.pythonhosted.org/packages/49/df/50aa1999ab9bde74656c2919d9c0c085fd2b3775fd3eca826012bef76d8c/requests-2.18.4-py2.py3-none-any.whl (88kB)
    100% |████████████████████████████████| 92kB 1.4MB/s 
Collecting docker-pycreds>=0.2.3 (from docker)
  Downloading https://files.pythonhosted.org/packages/5c/b6/3a4ca03225f6e9a25cb47cd3c695d19b88df5928ab9310e64dec1eb3b4d1/docker_pycreds-0.2.3-py2.py3-none-any.whl
Collecting websocket-client>=0.32.0 (from docker)
  Downloading https://files.pythonhosted.org/packages/8a/a1/72ef9aa26cfe1a75cee09fc1957e4723add9de098c15719416a1ee89386b/websocket_client-0.48.0-py2.py3-none-any.whl (198kB)
    100% |████████████████████████████████| 204kB 1.1MB/s 
Collecting six>=1.4.0 (from docker)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests!=2.18.0,>=2.14.2->docker)
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 1.4MB/s 
Collecting certifi>=2017.4.17 (from requests!=2.18.0,>=2.14.2->docker)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 1.2MB/s 
Collecting urllib3<1.23,>=1.21.1 (from requests!=2.18.0,>=2.14.2->docker)
  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 1.1MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests!=2.18.0,>=2.14.2->docker)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 1.3MB/s 
Installing collected packages: idna, certifi, urllib3, chardet, requests, six, docker-pycreds, websocket-client, docker
Successfully installed certifi-2018.4.16 chardet-3.0.4 docker-3.3.0 docker-pycreds-0.2.3 idna-2.6 requests-2.18.4 six-1.11.0 urllib3-1.22 websocket-client-0.48.0
(exec_run_ningx) # python3 
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
>>> containers = docker.DockerClient().containers
>>> container = "test_exec_run"
>>> # using local daemon
>>> [containers.get(container).exec_run('echo "hello"').output.decode('utf-8') for _ in range(1, 100)]
['hello\n', 'hello\n', 'hello\n', 'hello\n', 'hello\n', 'hello\n', 'hello\n', 'hello\n', 'hello\n', 
[...]
'hello\n', 'hello\n', 'hello\n']
>>> # using coreos daemon
>>> containers = docker.DockerClient(base_url="unix:////tmp/docker_core1.sock").containers
>>> [containers.get(container).exec_run('echo "hello"').output.decode('utf-8') for _ in range(1, 10)]
['hello\n', 'hello\n', 'hello\n', 'hello\n', '', 'hello\n', '', '', 'hello\n']
>>> 

Reading code of this project I'm wondering if that would be related to the daemon and not this lib but not sure where to create a proper issue.

Start at the library's exec_run entry point and reproduce the repeated echo command against both the local daemon and the CoreOS daemon through the SSH socket tunnel. Compare the returned output behavior and determine whether the defect is in this library or the Docker daemon; done means the cause is established and a focused regression test or actionable report exists.