docker / docker/cli

"docker pull" from Amazon ECR registry fails with HTTP 505 behind SonicWall NSA 4700

Open
#4,158 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

area/distribution area/networking kind/bug status/0-triage version/23.0
Dominant language
Go
Stars
6.1k
Forks
2.2k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Description

We use a hardware firewall in our company, which breaks up HTTPS traffic and injects its own SSL certificates. This way, downloaded files can be checked for viruses in advance. We were already able to solve the annoying issue with the self-signed SSL certificates without any problems using board tools.

Now, however, one problem remains: It seems that the firewall cannot handle HTTP/2 cleanly and performs a dirty fallback to HTTP/1.1. I suspect "docker pull" negotiates HTTP/2 with our Private ECR registry for the download, and then it is fallbacked to HTTP/1.1 uncleanly by the firewall. I'm not an expert on this.

Question: Is there any way to get "docker pull" to use only HTTPS with HTTP/1.1?

Tests with the firewall turned off have shown that the download then works. With the command

curl -sI https://curl.se -o/dev/null -w '%{http_version}\n'

we also proved that HTTP/1.1 is used when the firewall is active and HTTP/2 is used when the firewall is inactive. Hence my very strong suspicion that it is all related to HTTP/2.

I guess firewall virus scanners can't do much with HTTP/2 because of multiplexing and therefore always have to go back to HTTP/1.1.

I am grateful for any help!

Best regards
Tobias

Reproduce

$ docker pull XXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/project/repository
latest: Pulling from project/repository
d2acd4c3290a: Retrying in 1 second
error pulling image configuration: download failed after attempts=6: received unexpected HTTP status: 505 HTTP Version not supported

Expected behavior

Pulling the image from Amazon ECR-Registry should succeed.

docker version
Client: Docker Engine - Community
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.17.10
 Git commit:        aa7e414
 Built:             Thu May 12 09:17:39 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:46:54 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.19
  GitCommit:        1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., v2.16.0)
  scan: Docker Scan (Docker Inc., v0.23.0)

Server:
 Containers: 8
  Running: 8
  Paused: 0
  Stopped: 0
 Images: 124
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.10.0-14-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.41GiB
 Name: id-teams-03
 ID: NRU3:ASEV:ZSZG:DLKT:Y7VF:OWGP:77Q6:RV74:45E4:QWW4:SJFY:OQ4C
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 84
  Goroutines: 79
  System Time: 2023-04-04T08:26:16.910498617+02:00
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  nexus.id-berlin.local:9094
  nexus.id-berlin.local:9095
  nexus:9094
  nexus:9095
  127.0.0.0/8
 Registry Mirrors:
  http://nexus.id-berlin.local:9094/
 Live Restore Enabled: false
Additional Info

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the docker pull registry-download path and trace how its HTTP transport negotiates protocol when accessing Amazon ECR. Reproduce the 505 response with the SonicWall setup, then determine whether a supported HTTP/1.1 configuration or code change is possible; done means the pull succeeds in that environment or the limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
cli, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.