NginxProxyManager / NginxProxyManager/nginx-proxy-manager
504 Gateway Timeout randomly occurring
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 34.2k
- Forks
- 3.9k
- Avg merge
- 21h 12m
- Merged PRs (30d)
- 20
Description
Checklist:
Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
Yes
Are you sure you're not using someone else's docker image?
Yes
Have you searched for similar issues (both open and closed)?
Yes
Describe the bug:
I am able to use NPM successfully, and it works most of the time. Occasionally, almost at random (every day, a few times a day), the page will try to load and eventually show this message:
504 Gateway Time-out
openresty
I will restart the app and the db, then it works. When it does show the 504 Gateway Time-out page, I can go to the NPM login page, log in successfully, and interact with the dashboard and proxy host as if it is working fine.
I only have one proxy host redirecting to Nextcloud Hub. Nextcloud is in a separate VM, so it redirects based on IP address.
Scheme = html
Forward Hostname/IP = 192.168.100.33
Forward Port = 80
Nginx Proxy Manager Version - 2.9.18
To Reproduce - Steps to reproduce the behavior:
I wish I knew. After I restart the stack from the Portainer dashboard, it works. Then, depending on some unknown factor, it will time out. It may only take one minute, or a few hours. The logs always are the same, of which I can't see any errors or issues.
Expected behavior:
It does not time out.
Screenshots:
I am not sure what to provide.
Operating System:
Ubuntu Server 20.04.4
Docker 20.10.17
Docker Container
Portainer 2.13.1
Additional Context:
Docker Compose yml file:
version: '2'
services:
app:
image: 'jc21/nginx-proxy-manager:latest'
ports:
- '80:80'
- '81:81'
- '443:443'
environment:
DB_MYSQL_HOST: "db"
DB_MYSQL_PORT: 3306
DB_MYSQL_USER: "not-real"
DB_MYSQL_PASSWORD: "also-not-real"
DB_MYSQL_NAME: "npm"
volumes:
- ./data:/data
- ./letsencrypt:/etc/letsencrypt
healthcheck:
test: ["CMD", "/bin/check-health"]
interval: 10s
timeout: 3s
db:
image: 'jc21/mariadb-aria:latest'
environment:
MYSQL_ROOT_PASSWORD: 'not-real'
MYSQL_DATABASE: 'npm'
MYSQL_USER: 'this-is-not-real'
MYSQL_PASSWORD: 'again-not-real'
volumes:
- ./data/mysql:/var/lib/mysql
DB Logs from today prior to it timing out:
2022-07-26 22:57:48 3 [Warning] Aborted connection 3 to db: 'npm' user: 'npmuser' host: '172.18.0.3' (Got an error reading communication packets)
2022-07-26 22:57:54 0 [Note] /usr/bin/mysqld (initiated by: unknown): Normal shutdown
2022-07-26 22:57:54 0 [Note] Event Scheduler: Purging the queue. 0 events
2022-07-26 22:57:55 0 [Note] /usr/bin/mysqld: Shutdown complete
[i] pre-init.d - processing /scripts/pre-init.d/01_secret-init.sh
[i] mysqld already present, skipping creation
[i] MySQL directory already present, skipping creation
2022-07-26 22:58:06 0 [Note] /usr/bin/mysqld (mysqld 10.4.15-MariaDB) starting as process 1 ...
2022-07-26 22:58:06 0 [Note] Plugin 'InnoDB' is disabled.
2022-07-26 22:58:06 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-07-26 22:58:06 0 [Note] Server socket created on IP: '::'.
2022-07-26 22:58:06 0 [Warning] 'user' entry '@526f1f7a5d7d' ignored in --skip-name-resolve mode.
2022-07-26 22:58:06 0 [Warning] 'proxies_priv' entry '@% root@526f1f7a5d7d' ignored in --skip-name-resolve mode.
2022-07-26 22:58:06 0 [Note] Reading of all Master_info entries succeeded
2022-07-26 22:58:06 0 [Note] Added new Master_info '' to hash table
2022-07-26 22:58:06 0 [Note] /usr/bin/mysqld: ready for connections.
Version: '10.4.15-MariaDB' socket: '/run/mysqld/mysqld.sock' port: 3306 MariaDB Server
App logs from today prior to timing out:
[7/26/2022] [9:44:15 AM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[7/26/2022] [9:44:16 AM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4
[7/26/2022] [9:44:16 AM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6
[7/26/2022] [9:44:17 AM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [9:44:17 AM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [9:44:17 AM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [10:44:15 AM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [10:44:17 AM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [10:44:17 AM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [11:44:15 AM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [11:44:17 AM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [11:44:17 AM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [12:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [12:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [12:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [1:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [1:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [1:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [2:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [2:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [2:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [3:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [3:44:15 PM] [IP Ranges] › ℹ info Fetching IP Ranges from online services...
[7/26/2022] [3:44:15 PM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[7/26/2022] [3:44:16 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4
[7/26/2022] [3:44:16 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6
[7/26/2022] [3:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [3:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [3:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [4:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [4:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [4:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [5:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [5:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [5:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [6:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [6:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [6:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [7:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [7:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [7:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [8:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [8:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [8:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [9:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [9:44:15 PM] [IP Ranges] › ℹ info Fetching IP Ranges from online services...
[7/26/2022] [9:44:15 PM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[7/26/2022] [9:44:16 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4
[7/26/2022] [9:44:16 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6
[7/26/2022] [9:44:16 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [9:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [9:44:17 PM] [SSL ] › ℹ info Renew Complete
[7/26/2022] [10:44:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [10:44:17 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [10:44:17 PM] [SSL ] › ℹ info Renew Complete
[cont-finish.d] executing container finish scripts...
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01_perms.sh: executing...
Changing ownership of /data/logs to 0:0
[cont-init.d] 01_perms.sh: exited 0.
[cont-init.d] 01_s6-secret-init.sh: executing...
[cont-init.d] 01_s6-secret-init.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
❯ Enabling IPV6 in hosts: /etc/nginx/conf.d
❯ /etc/nginx/conf.d/default.conf
❯ /etc/nginx/conf.d/include/force-ssl.conf
❯ /etc/nginx/conf.d/include/ssl-ciphers.conf
❯ /etc/nginx/conf.d/include/block-exploits.conf
❯ /etc/nginx/conf.d/include/assets.conf
❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
❯ /etc/nginx/conf.d/include/proxy.conf
❯ /etc/nginx/conf.d/include/ip_ranges.conf
❯ /etc/nginx/conf.d/include/resolvers.conf
❯ /etc/nginx/conf.d/production.conf
❯ Enabling IPV6 in hosts: /data/nginx
❯ /data/nginx/proxy_host/1.conf
[7/26/2022] [10:58:14 PM] [Migrate ] › ℹ info Current database version: 20211108145214
[7/26/2022] [10:58:14 PM] [Setup ] › ℹ info Logrotate Timer initialized
[7/26/2022] [10:58:14 PM] [Setup ] › ℹ info Logrotate completed.
[7/26/2022] [10:58:14 PM] [IP Ranges] › ℹ info Fetching IP Ranges from online services...
[7/26/2022] [10:58:14 PM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[7/26/2022] [10:58:15 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4
[7/26/2022] [10:58:15 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6
[7/26/2022] [10:58:15 PM] [SSL ] › ℹ info Let's Encrypt Renewal Timer initialized
[7/26/2022] [10:58:15 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[7/26/2022] [10:58:15 PM] [IP Ranges] › ℹ info IP Ranges Renewal Timer initialized
[7/26/2022] [10:58:15 PM] [Global ] › ℹ info Backend PID 240 listening on port 3000 ...
[7/26/2022] [10:58:18 PM] [Nginx ] › ℹ info Reloading Nginx
[7/26/2022] [10:58:19 PM] [SSL ] › ℹ info Renew Complete
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied Docker Compose configuration, the app and database logs, and /data/nginx/proxy_host/1.conf; compare the timing of 504 responses with Nginx reloads, database messages, and container restarts. Reproduce the intermittent timeout if possible and use the logs and container state to identify the failing component; done means the cause is isolated and the proxy remains reachable without restarting the stack.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, mariadb, nginx
- Domain
- backend, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100