hashicorp / hashicorp/nomad

Nomad with HTTP Proxy

Open
#25,334 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stage/needs-investigation stage/waiting-reply theme/driver/docker type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Nomad version

Output from nomad version
Nomad v1.4.2 (039d70eeef5888164cad05e7ddfd8b6f8220923b)

Operating system and Environment details

Rocky 9.4 on Bare metal server
Nomad client hosted on this server.
Server has a public interface and a private interface.

Issue

I have setup a squid proxy through which I proxy all my HTTP/HTTPS traffic from my containers to the internet.
My containers use the proxy without a problem using an environment variable HTTPS_PROXY=myserverip:port.
Right now I also want to proxify the HTTP/HTTPS coming from the server host to the internet.
So I deploy the env variables on the system using files /etc/environment and /etc/profile.
It works as intended, when trying to curl an external http/https website it goes through squid proxy except for the nomad process that doesn't use the proxy but uses the public interface.
During my tests I blocked HTTP/HTTPS traffic on the public interface, so when I tried to launch a new nomad job, the job failed because it could not pull the docker container image from the remote registry.
That is how I saw that the traffic was going through the public interface, to further confirm it I did a tcpdump on the public interface and I caught the traffic going to the remote registry.
With this I am sure that nomad ignores the HTTPS_PROXY variable.
Is there a specific configuration that I am unaware of ?

Reproduction steps

So I used the HTTP_PROXY and HTTPS_PROXY variables inside the files /etc/environment and /etc/profile to deploy my proxy variables and make them persist on the system.
/etc/profile

https_proxy=http://serverip:19000
http_proxy=http://serverip:19000

/etc/environment

export https_proxy=http://serverip:19000
export http_proxy=http://serverip:19000

After that I source these files to update the system configuration, and I also restart the nomad service.

source /etc/profile
source /etc/environment
systemctl restart nomad

After that I try to launch a nomad job with a docker image not in the local images.

Expected Result

The docker image is pulled by nomad and the job is running.

Actual Result

The docker image is not pulled from the remote registry.
When the docker pull is done in the CLI it goes through squid as intended.

Job file (if appropriate)
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The payload names no source files or tests. Start by checking how the Nomad client service receives proxy variables when it pulls a Docker image, then reproduce the failed pull with the systemd restart and blocked public HTTP/HTTPS traffic; done means the image pull uses the configured Squid proxy.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.