apache / apache/mina-sshd

Multiple sessions created by the same SSH client disconnected and timed out one after another at the same time.

Open
#901 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.1k
Forks
400
Avg merge
5d 1h
Merged PRs (30d)
1

Description

### Version

2.15.0

### Bug description

My use case is remote port forwarding . Both the client and server are implemented with Apache MINA SSHD. And multiple custom global requests are sent before sending the tcpip-forward request. I have discovered several abnormal issues:
After the SSH connection is established, the client sends 5 consecutive keepalive heartbeats to the server. However, no global keepalive packets are captured in the server's debug logs. Occasionally, the server receives 1 or 2 keepalive packets and then stops receiving any more randomly. This results in the SSH connection dropping repeatedly.
Multiple sessions created by the same client almost disconnect one after another within the same minute, and then all drop together again after a few minutes.
I implemented logic on the client side to re-establish a new connection once the session object is closed. Strangely, every reconnection succeeds on the first attempt, and none of the custom global requests get lost. The only faulty behavior occurs with the keepalive mechanism on the server side.
The server initiates a TCP connection to a random forwarded port every 20 seconds. If no keepalive response is received and the session remains open at this time, the client often fails to receive the incoming TCP connection notification.

### Below is the client-side code
Image

while(true){
try (ClientSession session =
SshUtils.getClientSession(
sameSshClientObj)) {
//send global request1
//send global request2
//send global request3
try (ExplicitPortForwardingTracker portForwardingTracker =
createPortForwardingTracker(session)) {
while(!session.isClosed()){
//sleep x seconds
}
}
}
}

### Below is the server-side code that initiates a connection to a random forwarded port every 20 seconds.
try (Socket testSocket = new Socket(host, port)) {
}

Then I frequently observe debug logs showing heartbeat timeouts on the client side, alongside idle timeout logs on the server side.

### Actual behavior

Frequent heartbeat timeouts on the client side, while the server fails to receive heartbeat requests.

### Expected behavior

The server can receive heartbeats normally and send corresponding responses.

### Relevant log output

```Shell
### Below is the client-side log
2026-06-16T18:04:23.410+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: signalAuthFailure(ClientSessionImpl[classic-tx-sh
anghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8418@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) type=SshException, signal
led=false, first=false: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.411+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: exceptionCaught(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8418@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822])[state=Opened] SshException:
Got 5 heartbeat requests without reply
2026-06-16T18:04:23.412+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] o.a.s.c.session.helpers.SessionTimeoutListener []: sessionException(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8418@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) SshException: Got 5 heartbea
t requests without reply
2026-06-16T18:04:23.412+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] c.s.p.t.d.u.SshClientCreator$DialerClientConnectionService []: sendHeartBeat(ClientSessionImpl[classic-
tx-shanghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8418@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) failed (SshException
) to send heartbeat #6 request=keepalive@tunnel.sensorsdata.cn: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.419+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: signalAuthFailure(ClientSessionImpl[classic-tx-sh
anghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8329@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) type=SshException, signal
led=false, first=false: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.419+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: exceptionCaught(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8329@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822])[state=Opened] SshException:
Got 5 heartbeat requests without reply
2026-06-16T18:04:23.419+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] o.a.s.c.session.helpers.SessionTimeoutListener []: sessionException(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8329@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) SshException: Got 5 heartbea
t requests without reply
2026-06-16T18:04:23.421+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] c.s.p.t.d.u.SshClientCreator$DialerClientConnectionService []: sendHeartBeat(ClientSessionImpl[classic-
tx-shanghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8329@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) failed (SshException
) to send heartbeat #6 request=keepalive@tunnel.sensorsdata.cn: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.681+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: signalAuthFailure(ClientSessionImpl[classic-tx-sh
anghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8106@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) type=SshException, signal
led=false, first=false: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.681+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: exceptionCaught(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8106@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822])[state=Opened] SshException:
Got 5 heartbeat requests without reply

### Below is the server-side log
2026-06-16T18:01:57.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/223.76.236.226:55728]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40266/40000 ms.
2026-06-16T18:05:07.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/113.57.11.210:47586]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40488/40000 ms.
2026-06-16T18:18:07.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/113.57.11.210:39636]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40263/40000 ms.
2026-06-16T18:27:27.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/58.48.59.226:46870]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40066/40000 ms.
2026-06-16T18:51:08.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/58.48.59.226:55670]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40655/40000 ms.
```

### Other information

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the client-side ClientSession and sendHeartBeat path, then compare it with SessionTimeoutListener and the server-side IdleTimeout behavior. Reproduce the issue using multiple sessions, custom global requests, and ExplicitPortForwardingTracker while collecting both client and server logs. Done means the cause of missing keepalive requests and coordinated disconnects is identified and the expected heartbeat and forwarding behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.