concourse / concourse/docker-image-resource

Cannot find the host of private registry when run in kubernetes

Open
#282 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs-validation
Dominant language
Go
Stars
162
Forks
250
PR merge metrics
No merged PRs in 30d

Description

I have a kubernetes cluster with a Concourse instance and a docker-registry instance deployed, both with the official stable helm chart. The docker registry exposes a k8s service of ClusterIP type. And I tried to run a job pushing an image to this registry.

I'm referencing the registry with the k8s service name, appended by port number, as documented:

resources:
  - name: go-k8s-docker
    type: docker-image
    source:
      repository: docker-registry:5000/go-k8s-docker

However, the job failed with an error message indicating that it cannot find the registry's host name:

The push refers to repository [docker-registry:5000/go-k8s-docker]
Get https://docker-registry:5000/v2/: dial tcp: lookup docker-registry on 10.4.0.30:53: no such host

I intercepted into the job-running container, did a curl, and it could not resolve the host either, the same as the job's error message:

root@322bb276-f86d-4c93-6b7d-28fbc5c68af4:/tmp/build/put# curl -v docker-registry:5000
* Rebuilt URL to: docker-registry:5000/
* Could not resolve host: docker-registry
* Closing connection 0
curl: (6) Could not resolve host: docker-registry

However, when I opened a shell into the worker pod (which is the 'host' from the job-running container's perspective), the curl succeeded with no problem:

root@concourse-worker-0:/# curl -v docker-registry:5000
* Rebuilt URL to: docker-registry:5000/
*   Trying 10.0.6.26...
* TCP_NODELAY set
* Connected to docker-registry (10.0.6.26) port 5000 (#0)
> GET / HTTP/1.1
> Host: docker-registry:5000
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Date: Mon, 14 Oct 2019 06:04:55 GMT
< Content-Length: 0
<
* Connection #0 to host docker-registry left intact

Seems that the container is not able to utilize its host's DNS server?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported setup with the Concourse worker and Kubernetes ClusterIP registry, then compare DNS resolution from the worker pod and the job-running container using the shown curl and docker push commands. Done means the job container resolves docker-registry:5000 and can push the image successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
devops, infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.