clj-commons / clj-commons/clj-ssh
Potential deadlock on hanging SocketInputStream.socketRead
- Dominant language
- Clojure
- Stars
- 241
- Forks
- 66
- PR merge metrics
- No merged PRs in 30d
Description
We recently bumped into an unresponsive app (with no logs or stack traces). After thread dumping the app, the culprit seems to be an initial hanging read (which is common problem, see https://stackoverflow.com/questions/28785085/how-to-prevent-hangs-on-socketinputstream-socketread0-in-java). The hanging thread could reasonably die there, but this https://github.com/hugoduncan/clj-ssh/blob/develop/src/clj_ssh/ssh.clj#L400 lock prevents any other requests from that point forward. Is there a way to avoid global locking on the known-hosts file?
Here's the relevant bit of the thread dump:
```
"qtp1869210328-19" #19 prio=5 os_prio=0 tid=0x00007fb998801800 nid=0x1b runnable [0x00007fb9750d6000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.jcraft.jsch.IO.getByte(IO.java:82)
at com.jcraft.jsch.Session.read(Session.java:918)
at com.jcraft.jsch.UserAuthNone.start(UserAuthNone.java:85)
at com.jcraft.jsch.Session.connect(Session.java:389)
at com.jcraft.jsch.Session.connect(Session.java:183)
at clj_ssh.ssh$fn__6855.invokeStatic(ssh.clj:118)
at clj_ssh.ssh$fn__6855.invoke(ssh.clj:115)
at clj_ssh.ssh.protocols$fn__6813$G__6780__6822.invoke(protocols.clj:4)
at clj_ssh.ssh$connect.invokeStatic(ssh.clj:401)
- locked <0x0000000704fe13b0> (a java.lang.Object)
at clj_ssh.ssh$connect.invoke(ssh.clj:397)
```
I could see around 50 threads waiting for lock <0x0000000704fe13b0> to be released. We had to restart the app. Thanks
Contributor guide
No contributing guide indexed for this repository
Research direction
Read src/clj_ssh/ssh.clj around connect at lines 397-401 and the linked known-hosts locking code; trace how JSch's Session.connect can remain in SocketInputStream.socketRead. Done means a blocked connection does not hold the global known-hosts lock or prevent other requests from proceeding. No test or exact replacement is specified, so first identify the relevant locking behavior and a reproducible check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clojure, java
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100