Hooks failing when behind nginx

Ok, so I'm pretty stumped and not even sure how to debug further. A few weeks ago my installs of bulldozer and policy-bot both stopped working. I did some digging and both are hitting webhook secret verification errors:

Sep 08 00:31:20 ghbots docker[70236]: {"level":"warn","rid":"daflem2oo5qg00eb9hs0","github_event_type":"status","github_delivery_id":"9ccc034c-ab1c-11f1-9df8-cee3806fedc2","error":"payload signature check failed","time":"2026-09-08T00:31:20.621397068Z","message":"Received invalid webhook headers or payload"}

I've been using my policy-bot instance to test. I've tried the following:

1. Nginx -> policy-bot via HTTP (does not work, with error above)
2. Policy-bot direct via DNAT (works)
3. Nginx -> policy-bot via HTTPS (works)

The debug logs weren't really telling me much, so I started using a simple Python script to do the HMAC checks ( https://gist.github.com/akerl/3a323441f8e21d22316eba495a9651bc , and the token is one I've been using for testing but isn't in production use ).

With that behind nginx, it doesn't matter if the proxy_pass uses HTTP or HTTPS. Either passes validation.

I've tried using my normal nginx config and also a barebones one:

server {
    server_name HOST;
    listen *:443 ssl;
    listen [::]:443 ssl;
    http2 on;

    ssl_certificate /etc/nginx/ssl/policybot.akerl.org.crt;
    ssl_certificate_key /etc/nginx/ssl/policybot.akerl.org.key;

    add_header X-XSS-Protection          "1; mode=block" always;
    add_header X-Content-Type-Options    "nosniff" always;
    add_header Referrer-Policy           "no-referrer-when-downgrade" always;
    add_header Content-Security-Policy   "default-src 'self' http: https: ws: wss: data: blob: 'unsafe-inline'; frame-ancestors 'self';" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;




    location / {
        proxy_pass                         http://172.17.0.2:8080;
        proxy_http_version                 1.1;
        proxy_cache_bypass                 $http_upgrade;

        proxy_set_header Upgrade           $http_upgrade;
        proxy_set_header Connection        $connection_upgrade;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header Forwarded         $proxy_add_forwarded;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_set_header X-Forwarded-Port  $server_port;

        proxy_connect_timeout              60s;
        proxy_send_timeout                 60s;
        proxy_read_timeout                 60s;

    }
}

and

server {
    server_name HOST;
    listen *:443 ssl;
    listen [::]:443 ssl;
    http2 on;

    ssl_certificate /etc/nginx/ssl/policybot.akerl.org.crt;
    ssl_certificate_key /etc/nginx/ssl/policybot.akerl.org.key;

    location / {
        proxy_pass                         http://172.17.0.2:8080;
    }
}

My next step is to try to build a debug copy of go-githubapp to see what the heck I'm missing, but opening this in case anybody else has ideas.

Start by reproducing the webhook through nginx over HTTP and HTTPS, comparing both with the working direct DNAT path and the Python HMAC check described in the issue. Inspect go-githubapp's webhook signature verification and request handling; done means a webhook forwarded through nginx validates successfully without weakening verification.