Default synchronous connection pool blocks forever?

We've been using clj-http version 3.5 for a while, and are very happy with it. But today, we ran into an issue we haven't seen before. A specific pattern of requests--I can't figure out yet how to isolate this into a unit test--hangs, in what appears to be a wait for a connection to become available from the PoolingHttpClientConnectionManager.

The thread's stack looks like this (with some elision--I am using default middleware):

        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f0ce9c70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:377)
        
       ...

        at clj_http.core$request.invoke(core.clj:412)
        at clj_http.core$request.invoke(core.clj:347)
        at clojure.lang.Var.invoke(Var.java:379)
        at clj_http.client$wrap_request_timing$fn__3085.invoke(client.clj:1030)
        at clj_http.client$wrap_async_pooling$fn__3093.invoke(client.clj:1063)
        
      ...

        at clj_http.client$request_STAR_.invoke(client.clj:1165)
        at clj_http.client$get.doInvoke(client.clj:1171)

There are no other threads actively requesting network content.

I enabled pool logging, and here's the end of the console log from the pool:

DEBUG: (PoolingHttpClientConnectionManager.java:320)  Connection [id: 2][route: {}->http://bit.ly:80] can be kept alive indefinitely 
DEBUG: (PoolingHttpClientConnectionManager.java:326)  Connection released: [id: 2][route: {}->http://bit.ly:80][total kept alive: 3; route allocated: 1 of 2; total allocated: 3 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:255)  Connection request: [route: {}->http://www.gov.mb.ca:80][total kept alive: 3; route allocated: 0 of 2; total allocated: 3 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:288)  Connection leased: [id: 3][route: {}->http://www.gov.mb.ca:80][total kept alive: 3; route allocated: 1 of 2; total allocated: 4 of 10] 
DEBUG: (DefaultHttpClientConnectionOperator.java:139)  Connecting to www.gov.mb.ca/198.163.15.25:80 
DEBUG: (DefaultHttpClientConnectionOperator.java:146)  Connection established 192.168.0.200:48939<->198.163.15.25:80 
DEBUG: (LoggingManagedHttpClientConnection.java:88)  http-outgoing-3: set socket timeout to 5000 
DEBUG: (PoolingHttpClientConnectionManager.java:320)  Connection [id: 3][route: {}->http://www.gov.mb.ca:80] can be kept alive for 5.0 seconds 
DEBUG: (PoolingHttpClientConnectionManager.java:326)  Connection released: [id: 3][route: {}->http://www.gov.mb.ca:80][total kept alive: 4; route allocated: 1 of 2; total allocated: 4 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:255)  Connection request: [route: {}->http://bit.ly:80][total kept alive: 4; route allocated: 1 of 2; total allocated: 4 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:288)  Connection leased: [id: 2][route: {}->http://bit.ly:80][total kept alive: 3; route allocated: 1 of 2; total allocated: 4 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:320)  Connection [id: 2][route: {}->http://bit.ly:80] can be kept alive indefinitely 
DEBUG: (PoolingHttpClientConnectionManager.java:326)  Connection released: [id: 2][route: {}->http://bit.ly:80][total kept alive: 4; route allocated: 1 of 2; total allocated: 4 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:255)  Connection request: [route: {s}->https://bitly.com:443][total kept alive: 4; route allocated: 0 of 2; total allocated: 4 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:288)  Connection leased: [id: 4][route: {s}->https://bitly.com:443][total kept alive: 4; route allocated: 1 of 2; total allocated: 5 of 10] 
DEBUG: (DefaultHttpClientConnectionOperator.java:139)  Connecting to bitly.com/67.199.248.15:443 
DEBUG: (DefaultHttpClientConnectionOperator.java:146)  Connection established 192.168.0.200:57998<->67.199.248.15:443 
DEBUG: (PoolingHttpClientConnectionManager.java:255)  Connection request: [route: {}->http://bit.ly:80][total kept alive: 4; route allocated: 1 of 2; total allocated: 5 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:288)  Connection leased: [id: 2][route: {}->http://bit.ly:80][total kept alive: 3; route allocated: 1 of 2; total allocated: 5 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:320)  Connection [id: 2][route: {}->http://bit.ly:80] can be kept alive indefinitely 
DEBUG: (PoolingHttpClientConnectionManager.java:326)  Connection released: [id: 2][route: {}->http://bit.ly:80][total kept alive: 4; route allocated: 1 of 2; total allocated: 5 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:255)  Connection request: [route: {s}->https://bitly.com:443][total kept alive: 4; route allocated: 1 of 2; total allocated: 5 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:288)  Connection leased: [id: 5][route: {s}->https://bitly.com:443][total kept alive: 4; route allocated: 2 of 2; total allocated: 6 of 10] 
DEBUG: (DefaultHttpClientConnectionOperator.java:139)  Connecting to bitly.com/67.199.248.15:443 
DEBUG: (DefaultHttpClientConnectionOperator.java:146)  Connection established 192.168.0.200:57999<->67.199.248.15:443 
DEBUG: (PoolingHttpClientConnectionManager.java:255)  Connection request: [route: {}->http://bit.ly:80][total kept alive: 4; route allocated: 1 of 2; total allocated: 6 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:288)  Connection leased: [id: 2][route: {}->http://bit.ly:80][total kept alive: 3; route allocated: 1 of 2; total allocated: 6 of 10] 
DEBUG: (LoggingManagedHttpClientConnection.java:88)  http-outgoing-2: set socket timeout to 5000 
DEBUG: (PoolingHttpClientConnectionManager.java:320)  Connection [id: 2][route: {}->http://bit.ly:80] can be kept alive indefinitely 
DEBUG: (PoolingHttpClientConnectionManager.java:326)  Connection released: [id: 2][route: {}->http://bit.ly:80][total kept alive: 4; route allocated: 1 of 2; total allocated: 6 of 10] 
DEBUG: (PoolingHttpClientConnectionManager.java:255)  Connection request: [route: {s}->https://bitly.com:443][total kept alive: 4; route allocated: 2 of 2; total allocated: 6 of 10] 

I was able to work around the problem by wrapping the code related to this scenario with a (with-connection-pool ...) wrapper. I just wonder why this was necessary. Never seen this before. It isn't obvious that there's more than one outgoing request when the thread blocks indefinitely.

I do wonder about the Connection [id: 2][route: {}->http://bit.ly:80] can be kept alive indefinitely. Are these connections not being released?

For what it's worth, there was no difference if I tried clj-http 3.7.0.

Thanks,
Eric

Start at clj_http.core$request in core.clj:412 and 347, then trace the default middleware through client.clj:1030, 1063, and 1165. Compare the blocking request with the working with-connection-pool case and inspect the PoolingHttpClientConnectionManager logs. Done means identifying why the default pool can wait indefinitely and adding a reproducible regression test if the behavior can be isolated.