nginx caches upstream DNS for api/plugin_daemon/web indefinitely — partial container restarts cause 502 Connection Refused
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
1.16.1
### Cloud or Self Hosted
Self Hosted (Docker)
### Steps to reproduce
**Title:** nginx caches upstream DNS for `api`/`plugin_daemon`/`web` indefinitely — partial container restarts cause 502 Connection Refused
### Description
`docker/nginx/conf.d/default.conf.template` uses static `proxy_pass http://api:5001;`
(and similarly for `plugin_daemon`, `web`) in most `location` blocks. Nginx resolves
the Docker-internal hostname to an IP only once, at process startup, and caches it
for the container's lifetime.
If `api` or `plugin_daemon` are restarted independently of `nginx` (selective
`docker compose up -d api`, image update via Watchtower/Diun, etc.), Docker may
assign the freed IP to a different container in the same network (e.g. `worker`,
`worker_beat`). Nginx keeps forwarding to the stale cached IP, which no longer
belongs to the intended service, causing a 502.
### Steps to reproduce
1. Start the full Dify stack (tested on self-hosted 1.16.1).
2. `docker restart ` (without restarting nginx).
3. If the freed IP gets reassigned to a different container on the same network
before `api` reclaims it (e.g. `worker`/`worker_beat` starting/restarting around
the same time), any request through nginx to `/v1`, `/console/api`, `/e/`, etc.
returns 502.
### Expected behavior
nginx should re-resolve the upstream hostname periodically (as it already does for
`/socket.io/`) so a container restart doesn't leave a stale cached IP.
### Logs
2026/08/10 10:11:47 [error] 20#20: *386 connect() failed (111: Connection refused) while connecting to upstream, client: 172.22.0.1, server: _, request: "POST /e/xxxxxxxx/chat/completions HTTP/1.1", upstream: "http://172.22.0.9:5002/e/xxxxxxxx/chat/completions", host: "xxx.xxx.xxx.xxx:8090"
Client-facing:
172.22.0.1 - - [10/Aug/2026:13:00:05 +0000] "POST /v1/chat-messages HTTP/1.1" 502 157 "-" "n8n" "-"
### Root cause & fix
`/socket.io/` already avoids this correctly:
```nginx
location /socket.io/ {
resolver 127.0.0.11 valid=30s ipv6=off;
set $socket_io_upstream ${NGINX_SOCKET_IO_UPSTREAM};
proxy_pass http://$socket_io_upstream;
...
}
```
Applying the same `resolver + variable` pattern to `/console/api`, `/api`, `/v1`,
`/openapi`, `/files`, `/explore`, `/e/`, `/`, `/mcp`, and `/triggers` fixes it.
Tested locally on 1.16.1: confirmed 502s stop occurring after restarting `api`
independently, validated with `nginx -t` and a live functional test. Happy to open
a PR with the fix if useful — full corrected `default.conf.template` available on request.
### ✔️ Expected Behavior
nginx should re-resolve the upstream hostname periodically (as it already does
for /socket.io/ via `resolver 127.0.0.11 valid=30s`), so that restarting `api`
or `plugin_daemon` independently of `nginx` doesn't leave a stale cached IP
pointing at the wrong container.
### ❌ Actual Behavior
nginx resolves the Docker-internal hostname for `api`/`plugin_daemon`/`web` only
once, at process startup, and caches it for the container's lifetime (static
`proxy_pass http://api:5001;` with no resolver/variable). If the freed IP gets
reassigned to a different container on the same network before `api`/`plugin_daemon`
reclaim it (e.g. `worker`/`worker_beat` starting around the same time), nginx keeps
forwarding traffic to the stale IP, producing:
connect() failed (111: Connection refused) while connecting to upstream
...and a 502 response to the client (observed via n8n workflow calls to
/v1/chat-messages and LibreChat's OpenAI-compatible integration hitting /e/).
Contributor guide
Research direction
Start in docker/nginx/conf.d/default.conf.template, comparing the existing /socket.io/ resolver and variable pattern with the static proxy_pass locations for api, plugin_daemon, and web. Apply the same behavior to the listed routes, then run nginx -t and a live test that restarts api independently. Done means nginx periodically re-resolves the service hostnames and requests no longer return 502 after container restarts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, nginx
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100