NginxProxyManager / NginxProxyManager/nginx-proxy-manager

504 Gateway Timeout randomly occurring

Open
#2,173 15 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
TypeScript
Stars
34.2k
Forks
3.9k
Avg merge
21h 12m
Merged PRs (30d)
20

Description

Checklist:
Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
Yes
Are you sure you're not using someone else's docker image?
Yes
Have you searched for similar issues (both open and closed)?
Yes

Describe the bug:
I am able to use NPM successfully, and it works most of the time. Occasionally, almost at random (every day, a few times a day), the page will try to load and eventually show this message:

504 Gateway Time-out
openresty

I will restart the app and the db, then it works. When it does show the 504 Gateway Time-out page, I can go to the NPM login page, log in successfully, and interact with the dashboard and proxy host as if it is working fine.

I only have one proxy host redirecting to Nextcloud Hub. Nextcloud is in a separate VM, so it redirects based on IP address.
Scheme = html
Forward Hostname/IP = 192.168.100.33
Forward Port = 80

Nginx Proxy Manager Version - 2.9.18

To Reproduce - Steps to reproduce the behavior:
I wish I knew. After I restart the stack from the Portainer dashboard, it works. Then, depending on some unknown factor, it will time out. It may only take one minute, or a few hours. The logs always are the same, of which I can't see any errors or issues.

Expected behavior:
It does not time out.

Screenshots:
I am not sure what to provide.

Operating System:
Ubuntu Server 20.04.4
Docker 20.10.17
Docker Container
Portainer 2.13.1

Additional Context:

Docker Compose yml file:

version: '2'
services:
app:
image: 'jc21/nginx-proxy-manager:latest'
ports:
- '80:80'
- '81:81'
- '443:443'
environment:
DB_MYSQL_HOST: "db"
DB_MYSQL_PORT: 3306
DB_MYSQL_USER: "not-real"
DB_MYSQL_PASSWORD: "also-not-real"
DB_MYSQL_NAME: "npm"
volumes:
- ./data:/data
- ./letsencrypt:/etc/letsencrypt
healthcheck:
test: ["CMD", "/bin/check-health"]
interval: 10s
timeout: 3s
db:
image: 'jc21/mariadb-aria:latest'
environment:
MYSQL_ROOT_PASSWORD: 'not-real'
MYSQL_DATABASE: 'npm'
MYSQL_USER: 'this-is-not-real'
MYSQL_PASSWORD: 'again-not-real'
volumes:
- ./data/mysql:/var/lib/mysql

DB Logs from today prior to it timing out:

2022-07-26 22:57:48 3 [Warning] Aborted connection 3 to db: 'npm' user: 'npmuser' host: '172.18.0.3' (Got an error reading communication packets)
2022-07-26 22:57:54 0 [Note] /usr/bin/mysqld (initiated by: unknown): Normal shutdown
2022-07-26 22:57:54 0 [Note] Event Scheduler: Purging the queue. 0 events
2022-07-26 22:57:55 0 [Note] /usr/bin/mysqld: Shutdown complete

[i] pre-init.d - processing /scripts/pre-init.d/01_secret-init.sh
[i] mysqld already present, skipping creation
[i] MySQL directory already present, skipping creation
2022-07-26 22:58:06 0 [Note] /usr/bin/mysqld (mysqld 10.4.15-MariaDB) starting as process 1 ...
2022-07-26 22:58:06 0 [Note] Plugin 'InnoDB' is disabled.
2022-07-26 22:58:06 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-07-26 22:58:06 0 [Note] Server socket created on IP: '::'.
2022-07-26 22:58:06 0 [Warning] 'user' entry '@526f1f7a5d7d' ignored in --skip-name-resolve mode.
2022-07-26 22:58:06 0 [Warning] 'proxies_priv' entry '@% root@526f1f7a5d7d' ignored in --skip-name-resolve mode.
2022-07-26 22:58:06 0 [Note] Reading of all Master_info entries succeeded
2022-07-26 22:58:06 0 [Note] Added new Master_info '' to hash table
2022-07-26 22:58:06 0 [Note] /usr/bin/mysqld: ready for connections.
Version: '10.4.15-MariaDB' socket: '/run/mysqld/mysqld.sock' port: 3306 MariaDB Server

App logs from today prior to timing out:

[7/26/2022] [9:44:15 AM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[7/26/2022] [9:44:16 AM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4
[7/26/2022] [9:44:16 AM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6
[7/26/2022] [9:44:17 AM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [9:44:17 AM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [9:44:17 AM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [10:44:15 AM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [10:44:17 AM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [10:44:17 AM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [11:44:15 AM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [11:44:17 AM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [11:44:17 AM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [12:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [12:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [12:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [1:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [1:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [1:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [2:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [2:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [2:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [3:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [3:44:15 PM] [IP Ranges] › ℹ info Fetching IP Ranges from online services...
[7/26/2022] [3:44:15 PM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[7/26/2022] [3:44:16 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4
[7/26/2022] [3:44:16 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6
[7/26/2022] [3:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [3:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [3:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [4:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [4:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [4:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [5:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [5:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [5:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [6:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [6:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [6:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [7:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [7:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [7:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [8:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [8:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [8:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [9:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [9:44:15 PM] [IP Ranges] › ℹ info Fetching IP Ranges from online services...
[7/26/2022] [9:44:15 PM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[7/26/2022] [9:44:16 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4
[7/26/2022] [9:44:16 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6
[7/26/2022] [9:44:16 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [9:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [9:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [10:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [10:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [10:44:17 PM] [SSL ] › ℹ info Renew Complete
[cont-finish.d] executing container finish scripts...
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01_perms.sh: executing...
Changing ownership of /data/logs to 0:0
[cont-init.d] 01_perms.sh: exited 0.
[cont-init.d] 01_s6-secret-init.sh: executing...
[cont-init.d] 01_s6-secret-init.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
❯ Enabling IPV6 in hosts: /etc/nginx/conf.d
❯ /etc/nginx/conf.d/default.conf
❯ /etc/nginx/conf.d/include/force-ssl.conf
❯ /etc/nginx/conf.d/include/ssl-ciphers.conf
❯ /etc/nginx/conf.d/include/block-exploits.conf
❯ /etc/nginx/conf.d/include/assets.conf
❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
❯ /etc/nginx/conf.d/include/proxy.conf
❯ /etc/nginx/conf.d/include/ip_ranges.conf
❯ /etc/nginx/conf.d/include/resolvers.conf
❯ /etc/nginx/conf.d/production.conf
❯ Enabling IPV6 in hosts: /data/nginx
❯ /data/nginx/proxy_host/1.conf
[7/26/2022] [10:58:14 PM] [Migrate ] › ℹ info Current database version: 20211108145214
[7/26/2022] [10:58:14 PM] [Setup ] › ℹ info Logrotate Timer initialized
[7/26/2022] [10:58:14 PM] [Setup ] › ℹ info Logrotate completed.
[7/26/2022] [10:58:14 PM] [IP Ranges] › ℹ info Fetching IP Ranges from online services...
[7/26/2022] [10:58:14 PM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[7/26/2022] [10:58:15 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4
[7/26/2022] [10:58:15 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6
[7/26/2022] [10:58:15 PM] [SSL ] › ℹ info Let's Encrypt Renewal Timer initialized
[7/26/2022] [10:58:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [10:58:15 PM] [IP Ranges] › ℹ info IP Ranges Renewal Timer initialized
[7/26/2022] [10:58:15 PM] [Global ] › ℹ info Backend PID 240 listening on port 3000 ...
[7/26/2022] [10:58:18 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [10:58:19 PM] [SSL ] › ℹ info Renew Complete

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied Docker Compose configuration, the app and database logs, and /data/nginx/proxy_host/1.conf; compare the timing of 504 responses with Nginx reloads, database messages, and container restarts. Reproduce the intermittent timeout if possible and use the logs and container state to identify the failing component; done means the cause is isolated and the proxy remains reachable without restarting the stack.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, mariadb, nginx
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.