docker / docker/docker-py

client.containers.run may leave a container in a CREATED state and raise an exception

Open
#2,816 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.2k
Forks
1.7k
Avg merge
13d 8h
Merged PRs (30d)
2

Description

Reference: https://github.com/docker/docker-py/blob/27e3fa3e2e2a198ae95399629c2d492368c95965/docker/models/containers.py#L810-L821

On a machine that is somewhat loaded where Docker API calls are timing out, it is possible that L811 succeeds, however the subsequent call to container.start() raises an APIError, for example, due to timeout. When the caller handles this exception, they are not expecting to have a container in a CREATED state and not RUNNING, especially when running with detach set to True. Even if the caller handles the exception, and implements an exponential backoff to retry the call to client.containers.run(...), they may end up with one or more containers in a CREATED state until they finally have one in a RUNNING state.

This is a bit of a race condition. Clearly, Docker handles run as create and start, but create may succeed while start fails.

The fix for this is to:

  1. Return the container created from L811 irrespective of whether container.start() succeeds. This way the caller can easily try to start the container themselves. A fix for this could look like this:
        try:
            container.start()
        finally:
            if detach:
                return container

In docker-cli, this solution would be akin to getting the container id per lines: https://github.com/docker/cli/blob/35f023a7c22a51867fb099d29006ef27379bc7fe/cli/command/container/run.go#L143-L150

  1. run needs to handle the errors from container.start() and tear down the CREATED but not-yet-started container only when auto-cleanup is True. This would be akin to docker-cli tearing down the CREATED container in the case that auto-remove is set to enabled, e.g. https://github.com/docker/cli/blob/35f023a7c22a51867fb099d29006ef27379bc7fe/cli/command/container/run.go#L165.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at docker/models/containers.py lines 810-821 and compare the referenced docker-cli run.go sections. Trace the create-then-start flow, including detach and auto-remove behavior. Done means a failed start does not leave an unexpected CREATED container: detached callers can receive the created container, while auto-cleanup removes it when appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.