Interaction of network timeouts and attach(stream=True) lead to undesirable behavior
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 13d 8h
- Merged PRs (30d)
- 2
Description
One can call APIClient.attach(cid, stream=True) to obtain an iterable of output chunks from a running container. This is super useful to stream output from a container in real time.
Under normal behavior, the iterable returned by attach(stream=True) is exhausted when the container stops.
However, APIClient establishes a default timeout of 60s on requests.
If a container doesn't send output for <timeout> seconds, an exception will be raised due to network timeout. I believe docker.types.daemon.CancellableStream.__next__ swallows this exception and turns it into StopIteration. The output iterable terminates and from the perspective of the caller of .attach(stream=True), it looks like output is "done" and the container has stopped.
This behavior is somewhat confusing: if I attach(stream=True) to a container, I want to see all of the container's output until the container state changes or I abort the attach() request.
I think the existing workarounds for this are:
- Increase the network timeout.
- Poll the container state after
attach()finishes and callattach()again if the container is still running.
1 has the side-effect of affecting all requests. I do not want infinite timeout on any API request because I want some API requests to error if they take too long.
2 can result in loss of output data, as there is a race between re-attach()ing and new output data being generated.
Ideally there would be a way to stream output data without data loss without requiring raising a timeout on all API requests.
I think it would be acceptable for attach() to reconnect automatically on timeout for user convenience. But if data loss could occur, I think it is important to be documented in the docstring for attach().
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at APIClient.attach and docker.types.daemon.CancellableStream.next; trace how request timeouts become StopIteration during streaming. Determine whether timeout reconnect behavior can preserve output, or whether the attach() docstring must document possible data loss; done when the chosen behavior is specified and covered by appropriate regression tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100