Proposal: support environment variable of DOCKER_CLIENT_RETRIES

Description

At our company, we use the Docker CLI in our CI/CD automation. Some frequently used commands are:

• docker pull
• docker build
• docker push
• docker run
• docker tag

The problem is, we have a target of having our CI/CD runs be 99.9% reliable, but docker commands fail from time to time. These failures are transient and varied. Some failures are due to Docker outages such as the one on April 22nd. Other times, these issues present as 409's or TLS errors that resolve themselves upon restarting our CI/CD pipeline. These transient errors prevent us from reaching our 99.9% reliability goal.

Posts on the web suggest that to reduce transient failures, people should simply wrap their docker <whatever> commands with retry logic. However, this would be onerous to do with every use of docker we have. This also is problematic for docker servers, because this logic is usually a simple bash loop that doesn't use exponential backoff. That means that hand-written retry logic will hammer docker servers when they're struggling.

Proposal

This issue proposes a feature where an environment variable of DOCKER_CLIENT_RETRIES could be supported. It would default to 1 retry for backwards compatibility. When more than 1 is set as the number of retries, it would use exponential backoff.

DOCKER_CLIENT_RETRIES would be similar to DOCKER_TLS_VERIFY, which is also a client configuration parameter. For those already using docker pull and other commands in multiple places like us, this would allow them to bump up their retries through a simple environment variable export, rather than locating and editing every individual command.

I'm not certain yet, but I believe many docker commands are being routed through the client constructed in cli/trust.go here. If that's the case, it would likely be low-lift to replace the client with go-retryable. This client is extremely hardened, being used every day under high load in projects like Hashicorp Vault. It also is maintained by Hashicorp, and therefore has many paid developers interested in its success.

Please let me know if this seems accurate and feasible. 😄 If so, I am happy to code it up myself! I'm also happy to edit this potential concept as desired by the maintainers to keep it in line with the project's approaches and goals.

Thanks! 😄

Start by reading cli/trust.go around line 134 and tracing how Docker CLI commands construct and use their client. Assess whether the proposed DOCKER_CLIENT_RETRIES setting belongs in that path and how exponential backoff would apply. Done means the proposal's retry configuration works across the listed commands without requiring per-command shell loops.