clj-commons / clj-commons/clj-ssh

Potential deadlock on hanging SocketInputStream.socketRead

Open
#55 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Clojure
Stars
241
Forks
66
PR merge metrics
No merged PRs in 30d

Description

We recently bumped into an unresponsive app (with no logs or stack traces). After thread dumping the app, the culprit seems to be an initial hanging read (which is common problem, see https://stackoverflow.com/questions/28785085/how-to-prevent-hangs-on-socketinputstream-socketread0-in-java). The hanging thread could reasonably die there, but this https://github.com/hugoduncan/clj-ssh/blob/develop/src/clj_ssh/ssh.clj#L400 lock prevents any other requests from that point forward. Is there a way to avoid global locking on the known-hosts file?

Here's the relevant bit of the thread dump:

```
"qtp1869210328-19" #19 prio=5 os_prio=0 tid=0x00007fb998801800 nid=0x1b runnable [0x00007fb9750d6000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.jcraft.jsch.IO.getByte(IO.java:82)
at com.jcraft.jsch.Session.read(Session.java:918)
at com.jcraft.jsch.UserAuthNone.start(UserAuthNone.java:85)
at com.jcraft.jsch.Session.connect(Session.java:389)
at com.jcraft.jsch.Session.connect(Session.java:183)
at clj_ssh.ssh$fn__6855.invokeStatic(ssh.clj:118)
at clj_ssh.ssh$fn__6855.invoke(ssh.clj:115)
at clj_ssh.ssh.protocols$fn__6813$G__6780__6822.invoke(protocols.clj:4)
at clj_ssh.ssh$connect.invokeStatic(ssh.clj:401)
- locked <0x0000000704fe13b0> (a java.lang.Object)
at clj_ssh.ssh$connect.invoke(ssh.clj:397)
```

I could see around 50 threads waiting for lock <0x0000000704fe13b0> to be released. We had to restart the app. Thanks

Contributor guide

No contributing guide indexed for this repository

Research direction

Read src/clj_ssh/ssh.clj around connect at lines 397-401 and the linked known-hosts locking code; trace how JSch's Session.connect can remain in SocketInputStream.socketRead. Done means a blocked connection does not hold the global known-hosts lock or prevent other requests from proceeding. No test or exact replacement is specified, so first identify the relevant locking behavior and a reproducible check.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure, java
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.