docker / docker/docker-py

exec_run unexpected empty output when container is running on a remote host

Open
#3,332 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.2k
Forks
1.7k
Avg merge
13d 8h
Merged PRs (30d)
2

Description

Hello
I am observing incorrect output from exec_run when done on a running container on a remote host. The container is running a standard ubuntu image.
Connection to remote host is over an ssh forwarded unix socket.

Minimal reproducible example
Setup ssh forwarding:
ssh -L /tmp/docker-b.sock:/var/run/docker.sock -NTt -o ServerAliveInterval=30 -o ServerAliveCountMax=10 $DOCKER_HOST_B &

Connect to remote docker and execute command:

>>> import docker
>>> c = docker.DockerClient(base_url="unix:///tmp/docker-b.sock", version="auto", timeout=60, max_pool_size=8)
>>> c.containers.get("42270b507ba3").exec_run("echo hello")
ExecResult(exit_code=0, output=b'hello\n')
>>> c.containers.get("42270b507ba3").exec_run("echo hello")
ExecResult(exit_code=0, output=b'')
>>> c.containers.get("42270b507ba3").exec_run("echo hello")
ExecResult(exit_code=0, output=b'')
>>> c.containers.get("42270b507ba3").exec_run("echo hello")

Troubleshooting
I ran strace on the python process from where I ran exec_run and observed the following,
For unexpected output:

sendto(9, "POST /v1.43/exec/0ab9f918306ddcfdb18b1e36dce81b9ec6ebb2f03e4a26108f5128f73766c8c4/start HTTP/1.1\r\nHost: localhost\r\nUser-Agent: docker-sdk-python/7.1.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: tcp\r\nContent-Type: application/json\r\nContent-Length: 31\r\n\r\n", 286, 0, NULL, 0) = 286
recvfrom(9, "HTTP/1.1 101 UPGRADED\r\nContent-Type: application/vnd.docker.multiplexed-stream\r\nConnection: Upgrade\r\nUpgrade: tcp\r\nApi-Version: 1.43\r\nDocker-Experimental: false\r\nOstype: linux\r\n", 8192, 0, NULL, NULL) = 177
recvfrom(9, "Server: Docker/24.0.5 (linux)\r\n\r\n\1\0\0\0\0\0\0\6hello\n", 8192, 0, NULL, NULL) = 47

For expected output:

sendto(10, "POST /v1.43/exec/832bad59596ae017de0428004e1e9073c2a0e5571ca1ac385df09a27763f6c67/start HTTP/1.1\r\nHost: localhost\r\nUser-Agent: docker-sdk-python/7.1.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: tcp\r\nContent-Type: application/json\r\nContent-Length: 31\r\n\r\n", 286, 0, NULL, 0) = 286
recvfrom(10, "HTTP/1.1 101 UPGRADED\r\nContent-Type: application/vnd.docker.multiplexed-stream\r\nConnection: Upgrade\r\nUpgrade: tcp\r\nApi-Version: 1.43\r\nDocker-Experimental: false\r\nOstype: linux\r\nServer: Docker/24.0.5 (linux)\r\n\r\n", 8192, 0, NULL, NULL) = 210
recvfrom(10, "\1\0\0\0\0\0\0\6", 8, 0, NULL, NULL) = 8
recvfrom(10, "hello\n", 6, 0, NULL, NULL) = 6

So it looks like when the HTTP header is split into two recv calls, the issue is observed.

Full strace output:
docker-issue-strace.log

Additional info
Python version: Python 3.10.12
Docker python SDK version: 7.1.0
Container image: ubuntu:22.04
Docker version:

Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~22.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.3
  Git commit:       24.0.5-0ubuntu1~22.04.1
  Built:            Mon Aug 21 19:50:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        
 runc:
  Version:          1.1.7-0ubuntu1~22.04.2
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the exec_run entry point and trace how the upgraded, multiplexed response is read when the HTTP header arrives across multiple recv calls. Reproduce with the SSH-forwarded Unix socket example and compare the strace cases. Done means repeated exec_run("echo hello") calls consistently return b'hello\n'.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.