langgenius / langgenius/dify

nginx caches upstream DNS for api/plugin_daemon/web indefinitely — partial container restarts cause 502 Connection Refused

Open Beginner friendly
#40,448 1 comment 1 reaction 0 assignees View on GitHub
🐞 bug
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.

### Dify version

1.16.1

### Cloud or Self Hosted

Self Hosted (Docker)

### Steps to reproduce

**Title:** nginx caches upstream DNS for `api`/`plugin_daemon`/`web` indefinitely — partial container restarts cause 502 Connection Refused

### Description
`docker/nginx/conf.d/default.conf.template` uses static `proxy_pass http://api:5001;`
(and similarly for `plugin_daemon`, `web`) in most `location` blocks. Nginx resolves
the Docker-internal hostname to an IP only once, at process startup, and caches it
for the container's lifetime.

If `api` or `plugin_daemon` are restarted independently of `nginx` (selective
`docker compose up -d api`, image update via Watchtower/Diun, etc.), Docker may
assign the freed IP to a different container in the same network (e.g. `worker`,
`worker_beat`). Nginx keeps forwarding to the stale cached IP, which no longer
belongs to the intended service, causing a 502.

### Steps to reproduce
1. Start the full Dify stack (tested on self-hosted 1.16.1).
2. `docker restart ` (without restarting nginx).
3. If the freed IP gets reassigned to a different container on the same network
before `api` reclaims it (e.g. `worker`/`worker_beat` starting/restarting around
the same time), any request through nginx to `/v1`, `/console/api`, `/e/`, etc.
returns 502.

### Expected behavior
nginx should re-resolve the upstream hostname periodically (as it already does for
`/socket.io/`) so a container restart doesn't leave a stale cached IP.

### Logs
2026/08/10 10:11:47 [error] 20#20: *386 connect() failed (111: Connection refused) while connecting to upstream, client: 172.22.0.1, server: _, request: "POST /e/xxxxxxxx/chat/completions HTTP/1.1", upstream: "http://172.22.0.9:5002/e/xxxxxxxx/chat/completions", host: "xxx.xxx.xxx.xxx:8090"

Client-facing:
172.22.0.1 - - [10/Aug/2026:13:00:05 +0000] "POST /v1/chat-messages HTTP/1.1" 502 157 "-" "n8n" "-"

### Root cause & fix
`/socket.io/` already avoids this correctly:
```nginx
location /socket.io/ {
resolver 127.0.0.11 valid=30s ipv6=off;
set $socket_io_upstream ${NGINX_SOCKET_IO_UPSTREAM};
proxy_pass http://$socket_io_upstream;
...
}
```
Applying the same `resolver + variable` pattern to `/console/api`, `/api`, `/v1`,
`/openapi`, `/files`, `/explore`, `/e/`, `/`, `/mcp`, and `/triggers` fixes it.
Tested locally on 1.16.1: confirmed 502s stop occurring after restarting `api`
independently, validated with `nginx -t` and a live functional test. Happy to open
a PR with the fix if useful — full corrected `default.conf.template` available on request.

### ✔️ Expected Behavior

nginx should re-resolve the upstream hostname periodically (as it already does
for /socket.io/ via `resolver 127.0.0.11 valid=30s`), so that restarting `api`
or `plugin_daemon` independently of `nginx` doesn't leave a stale cached IP
pointing at the wrong container.

### ❌ Actual Behavior

nginx resolves the Docker-internal hostname for `api`/`plugin_daemon`/`web` only
once, at process startup, and caches it for the container's lifetime (static
`proxy_pass http://api:5001;` with no resolver/variable). If the freed IP gets
reassigned to a different container on the same network before `api`/`plugin_daemon`
reclaim it (e.g. `worker`/`worker_beat` starting around the same time), nginx keeps
forwarding traffic to the stale IP, producing:

connect() failed (111: Connection refused) while connecting to upstream

...and a 502 response to the client (observed via n8n workflow calls to
/v1/chat-messages and LibreChat's OpenAI-compatible integration hitting /e/).

Contributor guide

Open the contributing guide

Research direction

Start in docker/nginx/conf.d/default.conf.template, comparing the existing /socket.io/ resolver and variable pattern with the static proxy_pass locations for api, plugin_daemon, and web. Apply the same behavior to the listed routes, then run nginx -t and a live test that restarts api independently. Done means nginx periodically re-resolves the service hostnames and requests no longer return 502 after container restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, nginx
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.