nginx / nginx/agent

agent fails to communicate with nginx listen sockets with proxy protocol enabled

Open
#978 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
371
Forks
107
Avg merge
1d 16h
Merged PRs (30d)
37

Description

Bug Overview

Agent is unable to communicate with nginx listeners that are configured with proxy protocol enabled, e.g. in a configuration like this:

server {
    listen [::]:8888 default_server ipv6only=on proxy_protocol;
    listen 8888 default_server proxy_protocol;

    [..]
    location = /stub_status {
        stub_status on;
    }

    location /api {
        api write=on;
    }
    [..]
}
Expected Behavior

Depending on a scenario, the following behavioral alternatives could be expected:

  1. If there are other listeners available to reach out to API/status endpoints, use those instead.
  2. If there are no other listeners, agent would seed proxy protocol payload while talking to specific sockets e.g. like curl does with --haproxy-protocol option (https://curl.se/docs/manpage.html#--haproxy-protocol).
Steps to Reproduce the Bug
  1. Set up nginx or nginx-plus with proxy_protocol listeners (see the above example).
  2. Install the agent.
  3. Run the agent.
  4. Check the nginx error log for entries like these:
2025/02/11 15:23:32 [error] 3881802#3881802: *889720 broken header: "GET /api HTTP/1.1" while reading PROXY protocol, client: ::1, server: [::]:8888
2025/02/11 15:23:32 [error] 3881803#3881803: *889721 broken header: "GET /api HTTP/1.1" while reading PROXY protocol, client: 127.0.0.1, server: 0.0.0.0:8888
2025/02/11 15:23:34 [error] 3881802#3881802: *889722 broken header: "GET /api HTTP/1.1" while reading PROXY protocol, client: ::1, server: [::]:8888
2025/02/11 15:23:34 [error] 3881802#3881802: *889723 broken header: "??/????I?In??Y?$?2??ↅK??? ?j,???p;ƫa?R?K?6Aijܜѝ?L^V?,?0?+?/̨̩??̪?????????$?(?#?'?" while reading PROXY protocol, client: ::1, server: [::]:8888
2025/02/11 15:23:34 [error] 3881803#3881803: *889724 broken header: "GET /api HTTP/1.1" while reading PROXY protocol, client: 127.0.0.1, server: 0.0.0.0:8888
2025/02/11 15:23:34 [error] 3881803#3881803: *889725 broken header: "??T;"%uB????(????#???9??," while reading PROXY protocol, client: 127.0.0.1, server: 0.0.0.0:8888
  1. Check agent log for entries like these:
time="2025-02-11T15:23:02Z" level=warning msg="Unable to perform Stub Status API GET request: Get \"http://[::1]:8888/stub_status\": EOF"
time="2025-02-11T15:23:02Z" level=warning msg="Unable to perform Stub Status API GET request: Get \"http://127.0.0.1:8888/stub_status\": EOF"
time="2025-02-11T15:23:02Z" level=warning msg="Unable to perform NGINX Plus API GET request: Get \"http://[::1]:8888/api\": EOF"
time="2025-02-11T15:23:02Z" level=warning msg="Unable to perform NGINX Plus API GET request: Get \"http://127.0.0.1:8888/api\": EOF"
Environment Details
  • Target deployment platform: EC2 VM, local VM
  • Target OS: Ubuntu 22.04.5 LTS
  • Version of this project or specific commit: nginx-agent 2.39.0~jammy
  • Version of any relevant project languages: irrelevant
  • nginx version: nginx-plus-r33-p2
Additional Context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the agent entry points that issue Stub Status and NGINX Plus API GET requests, then reproduce the failure with the shown nginx proxy_protocol listeners and compare the request behavior with curl's --haproxy-protocol option. Done means the agent can successfully reach the relevant endpoints under the selected behavior and the nginx logs no longer report broken PROXY protocol headers.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, nginx
Domain
api, backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.