docker / docker/docker-py

Interaction of network timeouts and attach(stream=True) lead to undesirable behavior

Open
#2,166 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.2k
Forks
1.7k
Avg merge
13d 8h
Merged PRs (30d)
2

Description

One can call APIClient.attach(cid, stream=True) to obtain an iterable of output chunks from a running container. This is super useful to stream output from a container in real time.

Under normal behavior, the iterable returned by attach(stream=True) is exhausted when the container stops.

However, APIClient establishes a default timeout of 60s on requests.

If a container doesn't send output for <timeout> seconds, an exception will be raised due to network timeout. I believe docker.types.daemon.CancellableStream.__next__ swallows this exception and turns it into StopIteration. The output iterable terminates and from the perspective of the caller of .attach(stream=True), it looks like output is "done" and the container has stopped.

This behavior is somewhat confusing: if I attach(stream=True) to a container, I want to see all of the container's output until the container state changes or I abort the attach() request.

I think the existing workarounds for this are:

  1. Increase the network timeout.
  2. Poll the container state after attach() finishes and call attach() again if the container is still running.

1 has the side-effect of affecting all requests. I do not want infinite timeout on any API request because I want some API requests to error if they take too long.

2 can result in loss of output data, as there is a race between re-attach()ing and new output data being generated.

Ideally there would be a way to stream output data without data loss without requiring raising a timeout on all API requests.

I think it would be acceptable for attach() to reconnect automatically on timeout for user convenience. But if data loss could occur, I think it is important to be documented in the docstring for attach().

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at APIClient.attach and docker.types.daemon.CancellableStream.next; trace how request timeouts become StopIteration during streaming. Determine whether timeout reconnect behavior can preserve output, or whether the attach() docstring must document possible data loss; done when the chosen behavior is specified and covered by appropriate regression tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.