Socket.IO Redis pubsub connection drops periodically — missing TCP keepalive
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
1.16.1
### Cloud or Self Hosted
Self Hosted (Docker)
### Steps to reproduce
1. Deploy with multi-replica API behind a cloud LB or K8s service
2. Use an external Redis (`externalRedis.enabled: true`)
3. Watch the api pod logs
### ✔️ Expected Behavior
No redis reconnect noise in logs. Socket.io events delivered reliably.
### ❌ Actual Behavior
Every 30s-60s the pubsub listener gets disconnected and reconnects:
Cannot receive from redis... retrying in 1 secs
Cannot receive from redis... retrying in 1 secs
Collaboration still works most of the time because reconnections succeed within 1-2s, but events sent during the reconnection window get dropped.
### Root cause
PR #39587 removed `socket_timeout` from `_build_redis_options()` to fix #39423 (complete collaboration breakage). That was correct — a read timeout on a blocking listener causes constant reconnection storms.
But now the `pubsub.listen()` loop is a purely idle TCP connection with no keepalive probes. Network middleboxes (cloud LBs, K8s services, Redis proxies) will silently close idle connections, and the Redis client only finds out on the next health check or when `pubsub.listen()` fails.
Meanwhile the regular Redis clients in `ext_redis.py` already have keepalive configured:
```python
socket_keepalive=dify_config.REDIS_KEEPALIVE,
socket_keepalive_options=socket_keepalive_options,
The config values (REDIS_KEEPALIVE, REDIS_KEEPALIVE_IDLE, etc) all exist — just not wired through to the Socket.IO RedisManager.
Contributor guide
Research direction
Start by inspecting _build_redis_options() and ext_redis.py, then trace how the Socket.IO RedisManager creates its pubsub connection. Compare the existing REDIS_KEEPALIVE settings with the options used by the listener; done means the listener uses the configured keepalive behavior and no longer drops during the described idle periods.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, redis
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100