apache / apache/shenyu

[BUG] When Admin is in cluster mode, the restart causes the gateway node to keep reconnecting by websocket

Open
#6,461 2 comments 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
Java
Stars
8.8k
Forks
3.1k
Avg merge
7d 1h
Merged PRs (30d)
85

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Current Behavior

1. admin和网关都是k8s部署
2. admin配置集群模式
3. 网关使用k8s本地域名进行ws连接
4. 使用websocket同步数据
5. 重启admin

网关日志:2026-07-29 14:29:31.078 INFO [] ,,,WebSocketConnectReadThread-121,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:29:31.078 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:30:31.079 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:30:31.082 INFO [] ,,,WebSocketConnectReadThread-124,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:30:31.082 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:31:31.082 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:31:31.085 INFO [] ,,,WebSocketConnectReadThread-127,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:31:31.085 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:32:31.085 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:32:31.089 INFO [] ,,,WebSocketConnectReadThread-130,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:32:31.089 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:33:31.089 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:33:31.092 INFO [] ,,,WebSocketConnectReadThread-133,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending sync msg
2026-07-29 14:33:31.093 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is successful.....
2026-07-29 14:34:31.094 INFO [] ,,,shenyu-shared_wheel_timer-4,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,shenyu bootstrap websocket namespaceId: 403dfe7c-e94c-4582-8e71-994e8356f961
2026-07-29 14:34:31.097 INFO [] ,,,WebSocketConnectReadThread-136,o.a.s.p.s.d.w.c.ShenyuWebsocketClient,websocket connection server[ws://shenyu-admin.base.svc.cluster.local/websocket] is opened, sending syn

Admin日志

2026-07-29 15:08:31.249 ERROR [] ,,,http-nio-0.0.0.0-80-exec-10,o.a.s.a.l.w.WebsocketCollector,websocket send result is exception: java.io.IOException: java.io.IOException: Broken pipe
1785308911249 2026-07-29T07:08:31.249Z 2026-07-29 15:08:31.249 WARN [] ,,,http-nio-0.0.0.0-80-exec-10,o.a.s.a.l.w.WebsocketCollector,websocket close on client[10.10.16.173]
1785308911249 2026-07-29T07:08:31.249Z
1785308911249 2026-07-29T07:08:31.249Z ... 37 common frames omitted
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlockInternal(WsRemoteEndpointImplBase.java:313)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:521)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:171)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1337)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1366)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1440)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketWrapperBase$OperationState.start(SocketWrapperBase.java:1051)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1632)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.IOUtil.write(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.SocketDispatcher.writev(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/sun.nio.ch.SocketDispatcher.writev0(Native Method)
1785308911249 2026-07-29T07:08:31.249Z Caused by: java.io.IOException: Broken pipe
1785308911249 2026-07-29T07:08:31.249Z at java.base/java.lang.Thread.run(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:57)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:60)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:152)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsFrameServer.notifyDataAvailable(WsFrameServer.java:164)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsFrameServer.doOnDataAvailable(WsFrameServer.java:184)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:85)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsFrameBase.processInputBuffer(WsFrameBase.java:130)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsFrameBase.processData(WsFrameBase.java:284)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsFrameBase.processDataText(WsFrameBase.java:484)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.server.WsFrameServer.sendMessageText(WsFrameServer.java:130)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsFrameBase.sendMessageText(WsFrameBase.java:390)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.pojo.PojoMessageHandlerWholeBase.onMessage(PojoMessageHandlerWholeBase.java:102)
1785308911249 2026-07-29T07:08:31.249Z at java.base/java.lang.reflect.Method.invoke(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.websocket.WebsocketCollector.onMessage(WebsocketCollector.java:203)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.service.impl.SyncDataServiceImpl.syncAllByNamespaceId(SyncDataServiceImpl.java:153)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178)
1785308911249 2026-07-29T07:08:31.249Z at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.DataChangedEventDispatcher.onApplicationEvent(DataChangedEventDispatcher.java:52)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.DataChangedEventDispatcher.onApplicationEvent(DataChangedEventDispatcher.java:99)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.websocket.WebsocketDataChangedListener.onSelectorChanged(WebsocketDataChangedListener.java:84)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.websocket.WebsocketCollector.send(WebsocketCollector.java:279)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.shenyu.admin.listener.websocket.WebsocketCollector.sendMessageBySession(WebsocketCollector.java:293)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendText(WsRemoteEndpointBasic.java:36)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendString(WsRemoteEndpointImplBase.java:192)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:250)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:266)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlockInternal(WsRemoteEndpointImplBase.java:324)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:589)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSession.java:789)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:266)
1785308911249 2026-07-29T07:08:31.249Z at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlockInternal(WsRemoteEndpointImplBase.java:326)
1785308911249 2026-07-29T07:08:31.249Z 2026-07-29 15:08:31.249 ERROR [] ,,,http-nio-0.0.0.0-80-exec-10,o.a.s.a.l.w.WebsocketCollector,websocket collection on client[10.10.16.173] error: java.io.IOException: java.io.IOException: Broken pipe

每次Admin准备写回数据,ws连接就关闭了

### Expected Behavior

正常连接到master

### Steps To Reproduce

1. admin和网关都是k8s部署
2. admin配置集群模式
3. 网关使用k8s本地域名进行ws连接
4. 使用websocket同步数据
5. 重启admin

### Environment

```markdown
ShenYu version(s): 2.7.0.3
```

### Debug logs

_No response_

### Anything else?

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the WebsocketCollector, WebsocketDataChangedListener, and SyncDataServiceImpl locations shown in the stack trace, then reproduce the Kubernetes cluster-mode restart sequence while watching gateway and admin logs. Trace how the gateway reconnects and how the admin selects the master session after restart. Done means the gateway remains connected to the master and no longer repeatedly reconnects or produces broken-pipe errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
backend, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.